The approach of this scoping review follows the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines (Multimedia Appendix 1) [35]. Our primary research question was “What are the current applications, performance, and limitations of transformer models in neuroimaging segmentation?”

The goal was the extraction of key themes within recent literature related to transformer use in neuroimaging segmentation that will guide future research and clinical applications. Only the literature starting from 2019 was considered, since the rapid evolution in the development of transformer models for medical imaging is a key recent development in this field.

We defined transformer models as DL architectures relying on self-attention mechanisms, capable of processing sequential data and capturing long-range dependencies. From the neuroimaging perspective, we considered those studies where these models were applied to different modalities of brain imaging, focusing on MRI due to its prevalence in neurological diagnostics.

Our review process followed a systematic search in 4 major databases: Scopus, IEEE Xplore, PubMed, and ACM Digital Library. We present a comprehensive review of methodologies, results, strengths, and limitations of the included studies to derive useful insights that bridge technical developments with their implications in the clinical domain of neuroimage segmentation.

Studies were retrieved on May 22, 2023, through searching the following databases: IEEE Xplore, ACM Digital Library, Scopus, and PubMed. The search was limited to 5 years, from 2019 to 2023, to prioritize recent research and consisted of search queries related to transformers such as “transformer,” “deep learning,” and “self-attention”; queries related to neuroimaging such as “brain,” “neuroimaging,” “MRI,” “CT,” and “PET”; and queries related to the medical field such as “health care,” “medical,” “health.”

This review only included papers whose primary purpose was on the use of transformers for the segmentation of neuroimages. Our search included journal papers, conference papers, and dissertations that focused on applying transformer models to imaging scans (eg, MRI and CT) of the human brain. We excluded all studies that (1) used transformers for the segmentation of nonneuroimaging, raw brain signals, or electroencephalogram data; (2) were not in English, review papers, conference abstracts, preprints, protocols, and conference abstracts; (3) focused on neuroimaging tasks other than segmentation (eg, classification and prediction); and (4) were published before 2019.

For this review, we used a 3-step study selection process. First, we used EndNote (Clarivate) to remove duplicate studies returned by our initial search. Next, 3 independent reviewers (MI, AA, and MA) screened the titles and abstracts of the remaining papers to exclude irrelevant studies. We then obtained full texts of the studies that passed the initial screening, and the same 3 reviewers (MI, AA, and MA) examined them against our predefined inclusion criteria. Any disagreements between reviewers during the screening processes were resolved through in-person discussions until a consensus was reached.

The data extraction for this review is done in Microsoft Excel by 2 independent groups of 2 reviewers (MI and AA and MA and OE) to share the workload for extraction and resolve conflicts between the groups. Disagreements in data extraction were resolved through consensus during face-to-face discussions. Data extractions fall into 3 broad categories: study characteristics, neuroimaging acquisition, and transformer features.

Synthesis of included data was done using a narrative approach. Descriptive texts, tables, and figures describe and show the summary and characteristics of the data. Microsoft Excel was used to manage and synthesize the data. First, we depict the characteristics of each included study concerning publication year, type of publication, and country of origin. Then, we describe the neuroimaging acquisition of these studies by including the imaging modality, dataset, dataset accessibility, and neurological condition. Finally, it explains the transformer architecture of the included studies: the number of parameters, transformer type, hybrid component, and the training and evaluation methodology used along with loss function, optimizer, and metrics.

This scoping review synthesized and analyzed publicly available research studies. No direct human participant research was conducted; therefore, approval from an institutional review board or a research ethics committee was unnecessary. There was no collection, use, or dissemination of personal data from human participants in this study. Data extracted and analyzed in this review had been sourced from published studies, previously subjected to ethical review processes as part of their original publication. The review followed ethics in research practices, in that the representation of the included studies was true to form, and the methodology for the selection of studies and extraction was transparent. No individual participant data were accessed or reported in this review; hence, privacy and confidentiality were ensured. Since this study did not involve direct contact with human participants, issues regarding informed consent and protection of privacy and compensation of participants were therefore not relevant in this study. No images or supplementary material showing identifiable information of any individual were used in this review.

A total of 1246 publications were retrieved from the initial search of the selected databases. In the first round of screening, 261 duplicates were identified and removed through the use of EndNote X9, leaving 985 publications remaining. In the second round, 761 publications were excluded through the analysis of their titles and abstracts against our predefined inclusion or exclusion criteria. The remaining 224 publications continued to the third round of screening, which included a detailed full-text read-through and resulted in the exclusion of 156 publications. Of the 1246 initial publications, only 68 studies met our criteria and were thereby included in this review. Figure 2 depicts the full screening process in more detail.

Table 1 depicts the characteristics and metadata of each included study, including the publication year, country, and type. Included studies ranged between 2019 and 2023. Over half of the included studies were published in 2022, followed by 32.84% (n=22) in 2023. Studies included peer-reviewed papers (n=48, 71.64%) and conference papers (n=19, 29.36%). The included studies spanned a total of 13 countries, with China being by far the largest contributor in this domain, representing 68.66% (n=46) of the total studies. Following China, we can find the United States (n=5, 7.46%), the United Kingdom (n=4, 5.97%), and India (n=3, 4.48%), with other countries contributing 1 paper apiece.

Table 2 depicts the different imaging modalities used, the datasets used, and the different neurological conditions across the included studies. The included studies included a range of 6 different modalities, with MRI being by far the most common with 88% (n=59), followed by CT with 10.45% (n=7), and the remaining modalities with 1 each. Over half of the included studies used only 1 dataset (n=40, 59.70%) for training and evaluation purposes, followed by 23.88% (n=16) using 2 datasets. Of the 44 unique datasets used across the included studies, 70.45% (n=31) are public or open-source datasets, and 29.55% (n=13) are private datasets obtained directly from medical institutions. Regarding the public dataset category, the brain tumor segmentation dataset (BraTS dataset) is by far the most widely used, with 58.21% (n=39) of total studies using its variants (including BraTS 2015, 2017, 2018, 2019, 2020, and 2021), followed by Medical Segmentation Decathlon and low-grade glioma-Kaggle with 5.97% (n=4) each. The main neurological condition of the included studies was the segmentation of brain tumors, with 71.64% (n=48) of studies conducting research specifically in this area.

The proposed neuroimage segmentation techniques used various artificial intelligence (AI) techniques. In this review, we focused on the deep transformer–based techniques that have gained more attention recently. From the proposed models, we can find transformer-based, CNN with transformer-based, and generative adversarial network with transformer-based techniques. At the same time, the methods based on TransBTS, TransUNet, SeinUNet, and U-Net with transformer are the most used models for neuroimage segmentation. Figure 3 illustrates these models in terms of architecture. Table 3 depicts the characteristics of transformer models used within the included studies. From Table 3, we can find that 58.21% (n=39) of the included studies did not explicitly report the number of parameters of their proposed models. Of the studies that did, however, the majority of the transformer models proposed had between 20 and 40 million parameters (n=10, 14.93%), followed by 1 and 19 million (n=8, 11.94%). A majority of studies implemented a 3D segmentation network (n=42, 62.69%), with 37.31% (n=25) being 2D. An overwhelming 85.07% (n=57) of included studies proposed transformer models that are hybrid, with only 14.93% (n=10) of them being standalone transformer models. ViT was the most used transformer architecture, with 55.22% (n=37) of studies using it as its main component. Another significant transformer model is the Swin transformer, with 20.89% (n=14), followed by TransUnet, with 5.97% (n=4). Of the 57 hybrid transformer models, 55 (96.49%) studies opted for a combination of CNN with their transformer, and of those 55 CNN-transformer models, 56.36% (n=31) were U-Net based, and 9.09% (n=5) were ResNet based. Both generative adversarial network (n=2, 3.51%) and autoencoders (n=2, 3.51%) were also combined with transformers.

Table 4 depicts the loss function used, the optimizer used, and the different evaluation methods used across each included study. The loss function was not mentioned in 11.94% (n=8) of the included studies. Of the studies that mentioned it, the most popular loss function is a combination of cross-entropy and Dice loss with 40.30% (n=27) of included studies, followed by Dice loss with 19.40% (n=13). Adam is the most used optimizer, with 47.76% (n=32) of included studies using it, followed by AdamW at 14.92% (n=10). However, the optimizer was not mentioned in 22.39% (n=15) of studies. In terms of evaluation, over half of the included studies used at least two evaluation metrics (n=34, 50.75%), followed by 1 metric (n=11, 16.42%). Of these evaluation metrics, the Dice score is by far the most used, with 94.03% (n=63) of all studies using it, followed by HD95, 52.24% (n=35), and sensitivity, 28.36% (n=19).

Transformers have revolutionized the area of neuroimage segmentation by offering unparalleled capabilities in modeling complex features in medical imaging. It has the ability to model both local and global information, which substantially improves the accuracy of segmentation and therefore becomes very useful in various neurological applications. As shown in Table 5, the common strengths of transformer-based techniques include a high mean Dice score, effective fusion of multimodal MRI, and robust performance across diverse and complex datasets. However, these models also have substantial limitations in terms of high computational and memory costs, sensitivity to small areas of tumors, and possible overfitting on smaller datasets.

The main purpose of this scoping review is to conduct a thorough investigation into the use of different transformer models in the field of neuroimaging, specifically segmentation. From the gathered data, it is clear that the use of transformers in neuroimaging experienced a great boost in research from 2021 to 2022, with over half of the included studies being published in 2022 compared to only 10 studies in 2021. It is also important to note that for the year 2023, only the studies up to May 22 were included; yet, this constitutes a total percentage of the included studies of 32.84% (n=22) and could very well be even higher when the whole year is considered.

From the studies included in this review, it is clear that MRI is by far the most popular image modality for applying transformer models to neuroimaging segmentation. This can be attributed to how common the use of MRI is in the diagnosis of neurological illnesses, especially for brain tumors [27], wherein it is able to provide functional, structural, and metabolic information [27] through the use of its different modalities (T1, T2, T1ce, and fluid-attenuated inversion recovery). MRI is particularly suitable for neuroimaging segmentation purposes because of the high spatial resolution and soft tissue contrast, both being critical for any form of precise segmentation it exhibits, since it is able to show good detailed visualization of structures in the brain and distinction between different tissues of the brain sizes [1,2].

Another reason for the popularity of MRI in the included studies is the availability of brain MRI scans sourced from the widely used BraTS datasets [101]. This yearly and open-source dataset contains a wide variety of different MRI modalities that are manually annotated, making it a very important resource for developing and benchmarking segmentation methods based on different transformer models. This is why it is no surprise that it is by far the most used dataset in the included studies.

When it comes to neurological conditions, a majority of included studies in this review focused on the use of transformers in brain tumor segmentation. This can be attributed to multiple factors, including the availability of MRI scans from the BraTS dataset that are specifically for brain tumor segmentation. Brain tumors are also highly prevalent among all ages and have a high fatality rate [102], making it a prime area for research into new methods of diagnosis and treatment. In addition, brain tumors are fairly complex and irregular in both location and shape [27], which makes manual segmentation a very tedious and time-consuming process that would benefit greatly from increased research into more automated methods for segmentation. The BraTS dataset is also a factor, as it provides a large variety of MRI scans that are specifically for brain tumor segmentation. Transformers are particularly useful for brain tumor segmentation due to their self-attention mechanism, which allows them to account for different variations in tumor characteristics, such as size and shape, during the segmentation process [26].

Most included studies proposed and developed models with 3D segmentation networks, specifically for 3D imaging data. In terms of neuroimaging, 3D scans are more common in part due to the 3D nature of MRI scans. Since MRI is the most common imaging modality used in neuroimaging, it makes sense that it is preferable to develop models for 3D imaging data in order to avoid the loss of information. Even though 3D models are typically more accurate for 3D imaging segmentation, they are computationally expensive [26], which is why some proposed models in the included studies chose to instead extract 2D slices from 3D imaging data. While this technique is suitable, reducing a 3D scan into 2D slices can lead to a degradation of volume and spatial characteristics native to 3D data [103].

CNN-transformer hybrid models were used far more than standalone transformer models in the included studies, specifically in the form of a U-Net and transformer combination. These combinations capitalize on the strengths of both CNN and transformers while minimizing their weaknesses. CNN is particularly useful in extracting local features and spatial information from the provided scans; however, it often struggles to capture long-range dependencies due to its small kernel size [26,27]. On the other hand, transformers are able to model these long-range dependencies due to their self-attention module, making them very useful for neuroimaging segmentation, especially in the case of brain tumors [27]. This is why most included studies opted to use the use of CNN to capture local features and transformers to capture global features to increase the performance of their models in the task of segmentation [102].

This scoping review provides an overview of the available research regarding the use of transformers in the context of neuroimaging segmentation. These findings underline important implications for future research and applications in this area.

It is a notable finding of this review that many studies apply transformers, specifically to brain tumor segmentation, which might hint at the potential of transformers in assisting diagnosis and treatment planning in this field. As shown here, transformers are well-suited for this task. However, further research is needed to assess the real-world clinical usefulness of transformers for brain tumor segmentation. While brain tumors are an important challenge, the focus on this single application at this level would seem indicative of the current lack of large and good-quality datasets in many other big neurological diseases and conditions, such as Alzheimer and Parkinson disease, and strokes. Making publicly available manually annotated datasets of different neurological conditions would motivate new research and developments on the application of transformers in this field. On top of this, the heavy reliance of studies on the BraTS dataset shows that there is a need to diversify datasets in order to validate different models correctly. Most of the included studies favored hybrid use by combining CNN and transformers, which illustrates the complementary strengths of these architectures for neuroimaging segmentation. Success in hybrid techniques shows that further exploration of novel integrations between transformers, CNNs, and other modules could become a promising direction to achieve better performances on more complex medical image analysis problems. Improved accuracy in neuroimaging segmentation, through the ability of transformer models to extract local and global features, allows for more accurate identification of neurological conditions such as brain tumors. This will provide earlier diagnosis and treatment. Moreover, automation with these models will save much time of the clinicians in performing manual segmentation so that they can concentrate on the care of patients and other important tasks. Treatment planning may also be improved with transformer models, where the potential for more accurate and consistent segmentation results helps a lot in this respect. Moreover, these models can also potentially be integrated into clinical workflows without much hassle by developing user-friendly interfaces and collaboration between AI researchers and clinicians to ensure these tools are adopted and effectively used in practice.

This scoping review has numerous key strengths with regard to the analysis of transformer applications in neuroimaging segmentation. First, it gives a broad overview of the fast-evolving field by capturing recent works from 2019 through 2023. Second, it allows focusing on current research so that the review reflects the state-of-the-art in transformer applications for medical imaging. It is a systematic approach, covering 4 major databases; hence, wide and comprehensive coverage of the literature reviewed. The inclusion of journal papers and conference papers facilitates a wide view of both consolidated and emergent research. Third, this review gives elaborate insights into various aspects of transformer use in neuroimaging: imaging modality, dataset, neurological condition, and metric for performance evaluation. This level of analysis provides rich information relevant to both researchers and practitioners within the field. Finally, the review’s focus on brain tumor segmentation, while a limitation in some respects, also serves as a strength by providing an in-depth look at transformer applications in a critical area of medical imaging with significant clinical implications.

While this scoping review offers a number of strengths, its limitations need to be acknowledged so as to strike a balance. First, the review was on transformers in neuroimaging segmentation alone, excluding other medical imaging tasks or organs. This narrow focus allows for an in-depth analysis of transformer applications in brain imaging but may not be representative of the full spectrum of use that transformers have seen in medical imaging. This limitation could be reduced by expanding the scope of future reviews to multiple organ systems or imaging tasks, giving a wider look at transformer applications in medical imaging.

Second, the review was focused on studies published in the English language, published from 2019 up to 2023. This narrowing was necessary, as most current works are favored in this novelty area of transformer use in medical imaging. In so doing, this review criterion may have left out important non-English language publications or early applications of transformers. This is likely a limitation in the representation of research trends worldwide. In this respect, future studies can be designed to include more languages, also extending the date range to capture more diverse sets of publications and track the evolution of transformer use in medical imaging over an extended period.

Third, the fact that 58.21% (n=39) of the works included in this review were based on the BraTS dataset introduces a certain bias in the domain toward the segmentation of brain tumors. Though it is a very critical area, it might not be useful to represent transformers completely for other neurological conditions. Future research needs to give more emphasis to developing and publicly releasing manually annotated datasets about more neurological conditions to address this limitation. This will further encourage diverse applications of transformers in neuroimaging and provide a wider understanding of the capability of transformers across different pathologies.

The review demonstrated a high dominance by studies from China, with 46 (68.66%) studies of the total (see Multimedia Appendix 2 for detailed analysis). This aligns with broader publication patterns in AI research where Chinese institutions contribute approximately 40% of global publications. While this distribution reflects documented trends in international research output, future reviews might benefit from more diversified search strategies to ensure comprehensive coverage of global research activities in this field.

Finally, no formal quality or risk-of-bias assessment of the included studies was performed. Although this represents a common approach when it comes to scoping reviews, this limits the degree to which strong conclusions can be drawn about the relative effectiveness of various approaches to transformers. Future systematic reviews or meta-analyses may involve quality assessments to support more robust evidence in terms of the efficacy of transformer models in neuroimaging segmentation.

These findings point to a variety of promising directions for future research on the application of transformers to neuroimaging segmentation. First, future studies should develop novel integrations between transformers, CNNs, and other advanced modules that will further improve performance for complex medical image analysis tasks. This might be achieved by investigating various hybrid models leveraging the strengths of transformers and more traditional DL methods. Second, the extension of transformer applications to more neurological conditions other than brain tumors, which would allow a wider grasp of the potential capability of transformers across different pathologies. More clinical applications are likely to follow from here. Third, the development of new transformer-based methods or their combination with emerging techniques like diffusion models could further improve efficiency and robustness for both 2D and 3D brain segmentation. Fourth, future studies shall be done to bridge the current limitations in dataset diversity. This may be in creating and publishing manually annotated datasets for a wider range of neurological conditions that can enable transformers to apply to neuroimaging in more diversified ways. Finally, the translation of research findings into clinical practice remains a high unmet need. This transition will require extensive validation of transformer models on diverse, real-world datasets and close collaboration between AI researchers and clinicians. Such collaboration could result in the development of more clinically relevant models and user-friendly interfaces, which would expedite the translation of these advanced technologies into routine clinical practice.

This scoping review has thoroughly investigated the applications of transformers in neuroimaging segmentation and discovered a highly evolving field with great potential. The results of this paper have shown that transformer models, especially combined with CNNs in hybrid architectures, are also very promising for the task of brain MRI segmentation. Some of the big advantages of transformers include the modeling of long-range dependencies in images through self-attention mechanisms while still being able to perform local feature extraction. Such a combination uniquely allows for more accurate and detailed segmentation in highly complex neurological pathologies, like brain tumors.

There is clearly a trend toward 3D transformer models and hybrid CNN-transformer architectures, dominated by ViT as the variant of transformer used most frequently. These approaches also obtain superior performance on benchmark datasets, such as brain tumor segmentation tasks. However, reliance on the BraTS dataset highlights a requirement for more diverse data sources to ensure that performance could be validated across more multiple neurological conditions.

While this is promising, there are still important issues in the field: high computational costs associated with transformer models, overfitting on smaller datasets, and validation in larger clinical settings. Another issue is the geographical concentration of research output that highlights the need for greater diversity in the origins of studies worldwide to improve the generalizability of findings.

The future prospect of transformer models will unlock the potential that neuroimaging segmentation demands. Refining both architectures and training methods and integration into clinical workflows, transformations may provide state-of-the-art for fast, accurate, and reproducible brain-MRI segmentation, hence advancing clinical diagnosis and evaluation techniques for a better outcome in regard to patients with neurological disorders.

Although transformers have shown great improvement in neuroimaging segmentation, much potential is yet to be realized. Future work will need to be focused on present limitations, the extension of applications across a wider range of neurological conditions, and narrowing the gap between research and clinical practice to ensure that transformers are a valuable and impactful technology in medical imaging analysis.