Currently, we live in an era where an abundance of data is being produced worldwide. While the term “big data” is generally used for predictive analytics, health care data can be considered “big data” by definition, as it is high in volume, velocity, variety, and veracity [1]. Big data is more suited to computational analysis, rather than traditional manual methods, and automating this analysis is an attractive proposition. The growing need to handle large datasets in the field of health care has led researchers to seek to leverage artificial intelligence (AI) as a means of automation [2]. The recent development of generative artificial intelligence (GenAI), particularly large language models (LLMs), has opened new frontiers in data handling [3]. In brief, GenAI uses advanced architectures, model context, and user prompts to recognize patterns in extensive data sets and generate original outputs. In the case of LLMs, this is done via transformer architectures, advanced neural networks designed to deliver next-token prediction [4].

LLMs are a versatile tool with the potential to transform health care research, but they also pose distinct challenges. As with other health care innovations, the risks and benefits of using LLMs should be weighed before implementation. Increasingly, there is a growing effort to develop strategies for the responsible use of AI. For example, leading journals do not accept papers with AI as an author, and NICE (National Institute of Health and Care Excellence) has guidelines on the use of AI in evidence generation [5,6]. Internationally, the European Union is expected to create the first AI law to be enforced in 2026, stratifying AI systems by risk level and regulating them accordingly [7]. Canada has also drafted legislation on AI [8]. Most laws focus on AI companies rather than individuals and have not yet taken effect. In addition to specific laws surrounding AI, it is crucial to comply with current unrelated but relevant laws, such as the General Data Protection Regulation (GDPR), in the interest of safety [9,10]. This will involve enhancing data security and clearly defining accountability [11,12].

The use of health care data is already bound by GDPR, and LLMs have been used in health care research applications ranging from the analysis of medical records and images to enhancing drug discovery and informing the formulation of new treatments [13-17]. Automated documentation could further assist clinical practice through use cases such as writing discharge summaries and personalized treatment or medication management plans [18,19]. This is particularly timely in the United Kingdom, where the government has pledged to embed AI throughout the National Health Service to support routine administration tasks [20]. There has also been a push toward automation methodologies in health and care research to deliver timely insights. This is particularly true in the field of evidence synthesis. However, the conversation and movement toward automation in this field have been ongoing for 2 decades with little to no progress [21-23].

As the speed of technological improvements accelerates, this can often outpace our ability to understand, assess, and mitigate concerns regarding AI [24]. Such concerns include reliability, accuracy, transparency, various ethical, security, and privacy concerns, as well as environmental concerns [25-27]. Such issues may be intrinsic to the LLM, reflecting its technical capabilities; extrinsic to the LLM, often relating to how it is used; or they may fall under both categories. When considering health care data, protection, security, and accuracy are paramount. As such, it is crucial to understand the views of individuals working in health care practice and research surrounding the use of LLMs. While research has been conducted to understand different population views, there has been no effort to cross-reference and triangulate these views. This is imperative to understand the landscape as a whole and promote multidisciplinary combative action. Thus, this umbrella review examines the concerns of health care professionals and researchers to identify areas for improvement and understand the implications for practice. It is anticipated that through the robust identification of issues, steps can be taken to mitigate concerns, instill confidence in users of AI, and that the use of AI will become more responsible.

We aimed to map the concerns associated with the use of LLMs in health care research and practice through the following objectives: (1) identify systematic reviews that report concerns of health care professionals and health care researchers, and (2) perform qualitative analysis of the findings using inductive and deductive thematic analysis.

Following a scoping search that confirmed the feasibility of this study, a protocol for the systematic review was developed and registered with PROSPERO (CRD420250640997) on February 24, 2025. No amendments were made to the information provided in the protocol. The umbrella review was conducted in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines and the synthesis without meta-analysis guidelines, and reported using the PRIOR (Preferred Reporting Items for Overviews of Reviews) checklist, PRISMA-S (Preferred Reporting Items for Systematic Reviews and Meta-Analyses - Search), and PRISMA 2020 Abstract checklist (Checklists 1-3) [28-31].

The SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research Type) framework was used to outline the inclusion criteria as follows:

We included systematic reviews published since January 2017 (the year before the first LLMs being introduced publicly to ensure robust coverage of dates), where an outcome of the review was concerns surrounding the use of GenAIs or LLMs in a health and social care research context. We considered systematic reviews as defined by the authors, provided the methodology followed a recognizable process (ie, searching, screening, data extraction, risk of bias, and synthesis). Preprints and nonpeer-reviewed papers were excluded (Textbox 1).

Searches were completed on February 26, 2025, and updated on February 25, 2026, in seven databases: (1) Ovid MEDLINE (R) and Epub Ahead of Print, In-Process, In-Data-Review and Other Non-Indexed Citations, daily and versions; (2) Ovid Embase; (3) Scopus; (4) Web of Science; (5) JBI Database of Systematic Reviews and Implementation Reports; (6) Cochrane Database of Systematic Reviews; and (7) Epistemonikos.

Study registry searches, purposeful searching of gray literature sources, and citation chaining were neither performed nor were authors independently contacted for further data.

A de novo search strategy was developed in MEDLINE (Ovid) using the phenomenon of interest and evaluation measures concepts of the SPIDER framework: (ethic* or concern* or raises questions or equality or equity or racial or discriminat* or EDI or (equity diversity and inclusion) or adversely or perpetuat* or persist* or bolster or pitfall* or controvers* or worry or barrier or impede or obstacle or limitation or hindrance or hurdle) AND ((LLM or large language model or GenAI or generative AI or ChatGPT or OpenAI or gpt or Gemini or DeepSeek or LlaMA or Falcon or Cohere or PaLM or Claude v1 or autoregressive language or encoder-decoder or decoder or transformer or prompt engineer) AND (research or academ*). The strategy was peer reviewed by an information specialist and translated into other databases as appropriate (Multimedia Appendix 1). A date limit of 2017 onward was applied to all searches, and results were limited to systematic reviews using built-in functions in each database. No language restrictions were applied.

The records from the databases were imported into Rayyan, a free online screening platform, and duplicates were removed [32]. The remaining systematic reviews were initially screened by title and abstract in duplicate (FY and PA or MF). Discrepancies were resolved by discussion, and HO provided a final judgment when a consensus could not be reached. Systematic reviews taken forward for full-text screening were independently screened by 2 reviewers (FY and PA or MF). Again, discrepancies were resolved by discussion.

The data extraction template was initially trialed on 4 systematic reviews selected at random. Further, 2 reviewers independently extracted data from 50% of the included studies (FY and PA or MF). Discrepancies were resolved via discussion or consultation with HO. The remaining studies were extracted by 1 reviewer (FY) with discussion when required.

The following data were extracted: first author, year, title, DOI (Digital Object Identifier), journal, country of the first author, health care or research field, the LLM assessed, number of included studies, included study designs, inclusion criteria, exclusion criteria, key concerns, population raising the concern, notable quotes, statistical methods, and declared limitations.

AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews) was used to assess risk of bias, including reporting bias. The 16 questions outlined in AMSTAR-2 were applied independently by 2 reviewers to each included study, following the guidelines for each question or domain included [33,34].

The narrative synthesis was conducted by a single reviewer (FY), using a thematic analysis approach with inductive and deductive coding [35]. No statistical synthesis or meta-analysis was conducted, and quantitative effect measures were neither appropriate nor available given the qualitative nature of concerns. Concerns from the data extraction tables were analyzed and inductively synthesized into codes. Each systematic review was deductively analyzed to determine whether these codes were present in the text. The codes were organized and synthesized into main themes. The main themes were subsequently organized into overarching themes. An analysis by population was also conducted, which recorded the number of systematic reviews for a particular population that highlighted each of the coded concerns.

The application of GRADE (Grading of Recommendations Assessment, Development, and Evaluation) to systematic reviews of qualitative research gives a measure of how well the findings reflect the phenomenon of interest and provides an indicator of certainty around the evidence. GRADE was applied to assess the depth and breadth of this study, providing an initial rating, downgrading domains (risk of bias, inconsistency, indirectness, imprecision, and publication bias), upgrading domains (large effect or dose response, if confounders would reduce the effect), and an end certainty rating. Assessment was conducted independently by 2 reviewers. Heterogeneity and sensitivity analysis were not assessed within this review.

In total, 449 records were identified from databases, as shown in the PRISMA flowchart (Figure 1).

Before screening, 63 duplicates were removed. The remaining 386 systematic reviews were initially screened by title and abstract. A total of 317 reviews that did not meet the inclusion criteria were excluded. The remaining 69 systematic reviews underwent an additional full-text screening. Further, 27 reports were excluded, most commonly due to concerns not being addressed, followed by topics being unrelated to health or social care, and not being a systematic review or being inaccessible behind a paywall (Figure 1 and Multimedia Appendix 2). A final total of 42 remaining systematic reviews were included in this umbrella review.

All studies identified were written in the English language. However, the country of the first author varied across the globe. Most systematic reviews originated from the United States (n=9) [10,36-43], followed by the United Kingdom (n=6) [44-49], Pakistan (n=4) [50-53], Australia (n=3) [12,54,55], Canada and Israel (n=2 each) [56-59], and 16 other countries with 1 systematic review each [9,11,60-73]. The most common year of publication was 2024, with 27 reviews, followed by 2023 with 9 reviews and 2025 with 6 reviews. No reviews were found before 2023 (Table 1).

The population raising concerns was as follows. Most commonly, it was researchers only (n=17) and other groups included clinicians (general; n=5), plastic surgeons (n=5), psychiatrists (n=4), and neurosurgeons (n=3). Less common were pediatricians, gastroenterologists, dermatologists, pathologists, cardiologists, ophthalmologists, orthopedics, and ICU nurses (n=1 each; Table 1).

The most common individual LLM was ChatGPT, with 15 systematic reviews focusing on this LLM alone [9,36,39,42,44,48,50,51,59-61,68,70-72]. Another 15 systematic reviews considered more than one LLM [10,12,37,38,40,45,46,49,54,56-58,62,65,67]. A smaller proportion of studies (n=12) examined LLMs within a broader context, such as AI in general [11,43,47,53,69,73], natural language processing [41,66], conversational agents [55,63], deep learning [52], and GenAI [64] (Table 1).

There was a positively skewed distribution of primary studies included in the systematic reviews. The IQR was between 19 and 83 (median 32; range 5‐315) studies. Further, 3 systematic reviews were classed as outliers as they examined higher numbers of primary studies, and a further 3 reviews did not share their sample size [39,58,65]. All systematic reviews were qualitative and did not perform a meta-analysis. Only 5.1% (n=2) formally tested for agreement between reviewers using Cohen κ (Multimedia Appendix 3 for full data extraction).

Due to poor reporting of included primary studies within the systematic reviews, we were unable to calculate the corrected covered area to identify any overlap between primary studies. For example, some reviews listed the number of included studies but only referenced a subset in among supporting references, which meant the primary studies could not be easily identified. However, from manual inspection, it appears that primary studies that have been clearly reported were only included in a single systematic review. In either case, findings should be interpreted with caution as overinflation may be present.

Regarding risk of bias in systematic reviews, the overall rating of reviews using AMSTAR-2 was either low (n=2, 4.8% reviews) [56,57] or critically low (n=40, 95.2% reviews) [9-11,36-55,58-73] (Table 1). A total of 40 (95.2%) reviews had a valid research question (Q1) [9-12,36-41,43-65,67-73], 24 (57.1%) reviews provided descriptions of their included studies (Q3) [9,11,36,38,40-42,44,46,50,51,54-57,60-62,64-66,68,70,72], and 27 (64.2%) reviews reported no conflicts of interest (Q16) [9-12,36-44,46-60,62,63,67-73].

However, only 6 (14.2%) reviews had a protocol (Q2) [41,46,56,57,62,68], 3 (7.1%) reviews justified their choice of included studies (Q8) [41,56,65], 10 (23.8%) reviews had a literature search strategy (Q4) [41,48,55-57,61,62,64,66,68], 1 (38%) review was completely double-screened (Q5) [10,36,40,41,46,48,49,55-57,61,64,68,70,71,73], 8 (19%) reviews were completely double data-extracted (Q6) [36,53,54,56,57,61,73], 1 (2.3%) review justified their exclusions (Q7) [46], 6 (14.2%) reviews had risk of bias assessment (Q9) [36,46,55-57,62], 6 (14.2%) reviews looked for funding disclosures of constituent studies (Q10) [9,36,43,49,53,73], 6 (14.2%) reviews accounted for risk of bias when interpreting results (Q13) [36,41,55-57,65], and 5 (11.9%) reviews addressed heterogeneity (Q14) [36,41,55,57,67]. No reviews completed a meta-analysis (Q11), a risk of bias assessment for a meta-analysis (Q12), or formally assessed for publication bias (Q15; Table 2).

Reviews were overall graded as either low-end certainty (n=9, 21.4%) [36,41,46,51,55-57,62,65], or very low-end certainty (n=33, 78.6%) [9-12,37-40,42-44,47-50,52-54,58-61,63,66-73]. As there were no randomized controlled trials, all reviews started as low certainty before downgrading or upgrading (initial rating).

In the downgrading domains, all studies had consistent results (inconsistency) [9-12,36-73] and most addressed the core question (indirectness) of this umbrella review (n=36, 85.7%) [9-12,36-40,42,44-46,48-51,53-62,64-68,70-73]. However, only 7 (16.6%) reviews assessed for risk of bias [36,46,52,55-57,73], 14 (33.3%) reviews directly addressed the research question (imprecision) [10,12,36,40,43,46,48,51,53,56,57,62,65,73], and 4 (9.5%) reviews considered publication bias [41,51,62,65].

In terms of upgrading domains, 8 (19%) reviews were considered as showing large effects [10,36,46,51,56,57,62,65], but only 1 (2.3%) review mentioned confounders [41]. None displayed a dose-response relationship, and as this criterion was not relevant to this review, it was not taken into account for the final scoring (Table 3).

κ statistics were calculated between the reviewers (FY and one of PA, MF, or HO). Significant agreement was seen for both AMSTAR-2 (0.92, indicating near-perfect agreement) and GRADE (0.75, indicating substantial agreement). The κ statistic was considered significant where it was 0.6 (substantial agreement) or higher (Table 4) [74].

Qualitative coding is a means of deriving descriptive tags to categorize data, which can then be used to generate themes. A total of 29 codes were generated from the synthesis (Multimedia Appendix 3), and seven themes emerged: (1) data quality and reliability; (2) transparency and reproducibility; (3) performance and capability; (4) technical and operational; (5) human interaction and social impact; (6) ethical, legal, and safety; and (7) costs. These could be grouped under 3 core themes: technical capability; ethical, legal, and societal; and costs. For most population groups, mentions of technical capability concerns were the greatest, followed by ethical, legal, and societal concerns, and then cost concerns (Table 5).

The 42 studies reflected that bias in training algorithms and datasets mirrors what is already known on the subject, with estimates that around a quarter of studies on LLMs show bias [40]. For sensitive topics, this may be reduced to around 15% [65]. Key demographic barriers were mentioned, including sex, race, culture, language, and religion [43,55,61]. Political biases were also highlighted in closed-source algorithms [12]. Specifically, racial biases affecting individuals of black ethnicity were mentioned [62]. Sexual discrimination, bias toward female doctors was highlighted, with AI recommending fewer female doctors than male doctors. In pediatric medicine, a key issue was a limited, fragmented, or total lack of standardized training sets for LLMs in genetic disorders [45]. Most genetic disorders are rare and disproportionately affect children, which may contribute to the incomplete training sets [75].

Furthermore, outdated and limited datasets were commonly mentioned, with the date cutoffs highlighted for various ChatGPT models. For example, ChatGPT 3.5 was pretrained until September 2021 only and did not incorporate information from the internet [11,76]. ChatGPT 4 was pretrained up until April 2023 only [46,59,73]. Obvious problems with this include a lack of recent knowledge, obsolete knowledge, or misalignment with current clinical guidelines [50,53]. Another study also acknowledged that hospital-specific protocols should be adhered to, which LLMs may not include in their outputs [9]. From clinical practice, trust guidelines may differ from general ones, particularly in the case of antimicrobials, as location affects the presence of different microbes [77].

Fabricated or fake references were highlighted in 4 of the reviews [40,50,51,54]. Constituent studies that examined image references were anatomically incorrect or fabricated in 81% of cases [9]. Model overfitting, where a model is trained too specifically so that it cannot generalize on unseen data, was explored in reviews that examine image-based medical applications. Radiologically, x-ray images may be prone to overfitting [52]. Dermatology, which relies heavily on pattern recognition, may also be prone to this phenomenon [66]. Disease management options often featured fabricated references in all versions of ChatGPT [59]. Furthermore, the potential for such references to mislead junior colleagues, including resident doctors, was mentioned [59]. Similarly, several studies mentioned that hallucinations were a cause of concern [39,43,51,53,58,64]. Some studies emphasized that it was easy to be convinced by hallucinations [51,64]. Other aspects addressed included that hallucinations were often confidently given by LLMs, especially when parameter settings that encourage more varied, random outputs are used [46]. Other factors that influenced hallucination generation were the quality of prompts, with prompts phrased as stories often causing hallucinations [58]. It was estimated in 1 review that 40% of discharge summaries written by AI have hallucinations [39].

Several studies highlight the difficulties with the transparency of LLMs, with some referring to the technology as a “black box” [37,43,51,56,69,73]. Ethical discussions are highlighted as a way to build transparency of models [10]. Profit-driven lack of transparency was only mentioned in 1 review, but another highlighted how the proprietary nature of LLMs can complicate openness and trust [53,62]. Another review emphasized that a lack of transparency can be ascribed to multiple stages, including the design and training of LLMs [65]. The same review also emphasized that care should be taken so that interventions aimed at increasing transparency do not inadvertently expose patient data. Leaked personal data can potentially also amplify bias through normalizing biased correlations as real patterns [65].

The issue of repeatability with the same or similar prompts was also isolated as a concern. Most constituent primary research papers were identified as prompt experiments [64]. However, Patil et al [37] reported that about a quarter of the research papers were found not to disclose their prompts. Another review underlined that when they were disclosed, prompts tended to be single, standalone 1-shot prompts [59]. The concept of biased prompts was raised, and prompt injection, where prompts are engineered to extract information or cause disruption for nefarious purposes, was also highlighted as a concern [36,43].

The reviews in this study broadly agreed that the performance of LLMs was comparable to or exceeded that of humans. Latest estimates of correct responses on neurology licensing examination questions revealed ChatGPT 4 had 85% accuracy compared to human performance at 73.8% [46]. Fracture detection rates with deep learning were also close to this figure at 83% [67]. Approximately 76.5% (32/42) of studies used human performance as a benchmark, and 50% (21/42) compared LLMs to human performance alone. It is essential to measure success on diverse tasks, and some reviews agreed that LLMs often struggle with complexity [11,39,45,69,72]. Lack of originality was also a commonly cited concern [11,36,44,51,61,70,72]. Most systematic reviews emphasized that LLMs needed human oversight, and several mentioned it should only be viewed as a “supplementary tool” [9,39,45,50,56,57,60].

Few studies explored technical and operational challenges. Further, 1 review cited a primary study that examined system crashes [48]. Another review highlighted slow response times and how this was a problem in robotics [58]. Similarly, a second review noted slower clinical workflows due to the verification process [53]. Aside from the speed of development outstripping research, it was noted that a lack of research was problematic [64]. Another review quoted that only 26% of studies used randomized controlled trial–type designs in assessing LLMs by users, suggesting that conclusions from research may not be completely reliable [55]. Additionally, 1 review called for rigorous validation in real-world settings, which was supported by another review raising gaps in validation as a concern [43,73].

There was a mix of views regarding empathy and LLMs. Some reviews stressed that LLMs’ empathy was good, although it was usually assessed through technical expert reports rather than by both users and experts. Only a minority of studies (n=10) used an explicit definition of the term “empathy” [46,54,55]. Others stated there was a general lack of empathy [11,39,50,64]. Only 1 study gave a mix of such opinions [47].

Furthermore, some reviews expressed that LLMs could cause potential damage to the doctor-patient relationship [38,44,46,62]. There was a large overlap between this finding and reviews that addressed deskilling of the workforce or impacts on the job market [11,44,46,62,65]. Additionally, a lack of a sustainable relationship between new technologies and the user was emphasized in robotics due to problems with “semantics, consistency, and interactiveness” [63]. Researchers raised concerns around challenges with linguistic complexity, such as understanding irony and sarcasm [43].

User acceptance was identified as an important factor in 2 reviews [46,67]. The importance of acceptance by various groups, including physicians, caregivers, and providers, was highlighted [63]. Potential mistrust of LLMs and humanization issues were highlighted in the mental health field [43,55]. Yet, public acceptance was viewed as very positive in 1 systematic review [40]. There was an acknowledgment that this area needs more research [48].

Further, 5 reviews emphasized that existing inequalities could become more entrenched with LLMs [40,47,62,65,70]. This is separate from concerns over bias, which could also contribute to the deepening of inequalities. Both of these events could perpetuate the other, whereby biased outputs could deepen inequalities, which in turn could lead to the introduction of further biases.

Almost all reviews mentioned ethical concerns as a potential problem with LLM use, with varying degrees of explanation on this topic. Accountability was addressed from a 2-fold perspective, with 4 reviews focusing on medicolegal accountability [11,44,59,72]. Others were focused on legitimacy and accountability in research [50,51]. Suggested solutions included clear guidelines on accountability [12,48,55]. Another potential solution was policy and regulations [40,65]. Related to this was the issue of academic integrity, alternatively phrased as “pedagogical risk” [9,11,12,44,62,65,68].

Most reviews raised privacy and security concerns regarding LLMs. The privacy problems highlighted by studies can be seen as a triad involving information leaks from embedded training examples, inferential disclosure, and insufficiently deidentified data [40]. There may be a trade-off between data utility and privacy [40,65]. The need to comply with existing laws, including GDPR, was emphasized by 3 systematic reviews [9,12,59]. Other studies acknowledged that further efforts were necessary, such as limiting personal data collection or conducting audits [39,40,55,73].

Only 9 reviews discussed obtaining consent to collect personal data [10,12,37,39,43,44,53,62,73]. This reflects the limited literature available on ensuring informed consent when using LLMs and highlights the fact that new technology is outpacing safety. Just 1 review sought to explain consent itself, which entails providing a full disclosure of the risks and benefits in such a way that the participant comprehends and agrees [10]. Further, 2 reviews suggested that protocols are needed to obtain consent [12,39]. Consent is crucial in anticipation of seen and unforeseen ethical issues with LLMs, and yet other systematic reviews chose to discuss LLMs as a way to streamline consent forms [37,44,62].

Safety may become a population-wide problem as well as an individual one. Mass-scale problems may become apparent with the potential of infodemics perpetuated by LLMs, according to some reviews [51,53,62,65,72]. The saturation of scientific literature with low-quality automated reviews, which may fuel infodemics, was also discussed [62]. In a pandemic context such as COVID-19, there could be even greater consequences [65].

Finally, the risk of self-propagating and uncontrolled evolution was described as “unknown” by 1 review in passing [44]. Self-propagating and uncontrolled evolution relates to the LLM’s ability to grow and change on its own. Considering that this could be very detrimental and have long-term impacts, this is important. While this possibility has been informally mentioned in technology circles for some time, there are only a small number of new reports in the literature examining this possibility, so this finding is expected.

None of the reviews examined more than one cost aspect, and most lacked depth of answers in relation to this topic. Costs were broadly environmental related to computational processing [43,58,65]. Given the scale of climate change and its importance to health, this is an area of great interest and should be explored further [78,79]. Other costs mentioned were financial costs incurred by users [40,60].

This umbrella review aimed to narratively synthesize the concerns that health care professionals and researchers face when using AI. A wide variety of concerns are raised, which overlap and interlink, consistently affecting multiple populations. The findings indicate 3 core areas of shared concern within the health care field: technical capability of AI; ethical, legal, and societal implications for use; and associated costs.

Much of the concern surrounding technical capability lies with data quality and reproducibility. For research to be as robust as possible, we must use AI for tasks that are appropriate, judicious, and sense-checked, as well as monitoring the functions and effects of LLMs [38,39]. A good starting point is to acknowledge where, how, and when AI was used. Research integrity policies, reporting guidelines, and audits will be crucial in meeting quality standards and enabling reproducibility [9,37,40,65,73]. Most of the reviews included here advocated for guidelines, although none explored direct examples or attempted to construct a guideline. This is expected, given that international and national guidelines are only gradually emerging. However, it could be argued that research should be a proactive driver for exploring these issues rather than a reactive one.

Ethical, legal, and societal implications were varied and broad-ranging. Concerns over hallucinations, bias, inequalities, and consent provide interesting and often deeply interrelated perspectives that triangulate with the technical capabilities of AI. For instance, bias that perpetuates “hallucinations” was commonly cited [61,62,72]. Efforts for transparency may inadvertently perpetuate bias, as implicit biases in models can exist even when they show no explicit biases [80]. This gives a false sense of security surrounding systems that are lacking in objectivity. The reuse of biased datasets from such models will worsen this problem, and further research is needed to explore these perpetuating cycles. Ultimately, transparency must be judged by humans who may reinforce biases inadvertently or fail to acknowledge problems because features are unreported or untested. Therefore, while we should strive for more transparency, particularly through legislation, guidance, and reporting, we should avoid labeling any LLM as completely “transparent.”

Some biases were thought to be pervasive, for instance, “social biases entrenched in data.” Specifically, female ophthalmologists were recommended less than one-third as often as their male counterparts [47]. Biases were present in topical issues; for example, “unfavorable attitudes” were described when ChatGPT was prompted to discuss topics such as climate change and Black Lives Matter [65]. Research is uncovering both implicit and overt biases toward already disadvantaged populations, which is why careful consideration of LLMs and their applications is necessary to prevent exacerbating existing health inequalities. Education surrounding considerations of equality, diversity, and inclusion when using LLMs may be helpful on an individual level. However, discriminatory biases may only be overcome by careful curation of training data underpinning the LLMs.

Some terms regarding LLMs could be revised for clarity and to destigmatize. Findings indicate the term “hallucination” for AI issues is considered unhelpful and stigmatizing for those with a psychiatric disorder, reflecting proposals in the literature for the term “AI misinformation” [64,81]. The term “hallucination” was originally used in computing science to refer to retained outputs even when artificial neural networks were pruned by removing some connections [82]. The term has evolved, first to positively describe tasks related to computer vision and improved facial recognition, and then to mean the generation of incorrect outputs in translation or object detection. Currently, it is used to mean incorrect LLM outputs produced with confidence [83]. As an overall term to describe confidently produced errors, AI “hallucination” may, on the surface, be useful. However, LLMs are not produced the same way as biological hallucination, which occurs in the absence of external stimuli [84]. By contrast, LLMs have external stimuli in the form of training data and prompts but still produce nonsensical outputs. This can be stigmatizing for those with a psychiatric disorder and is technically imprecise. AI generalization, fact fabrication, or stochastic parroting could be used as more distinct terms depending on the types of error seen [83].

Interestingly, while cost was a core theme, there was no definitive cost element that was unifying across reviews. Further explorations of cost concerns could inform cost-benefit analyses and full economic evaluations of AI use cases. We have seen such evaluations for AI-assisted health care technologies, but not regarding AI use in general health care practice and research [85].

The constituent systematic reviews comprised primary research papers that were heterogeneous. For example, different papers used different methods of evaluation in 1 review, ranging from surveys to response ratings and interview feedback [55]. Moreover, the authors used different methods to classify the accuracy of LLMs and did not adhere to standard formal procedures for assessment [40]. The quality of the included reviews was generally poor, and the extent of publication bias was unknown. Many of the poorly reported reviews included studies, meaning we were unable to determine the overlap between primary studies using the corrected covered area. Caution should be taken that different populations’ views of LLM limitations are not a complete representation of views of the broader population, and that findings may be overinflated due to the unknown overlap of primary studies.

Furthermore, thematic analyses have an element of subjectivity. Potential sources of bias include researcher bias and confirmation bias, whereby preexisting beliefs and experiences may have influenced the coding. We have attempted to limit this through group discussion of codes; however, future studies could incorporate a blinded dual coding process.

This review is also limited by the search dates of the included systematic reviews. As the field of LLMs rapidly evolves, primary studies published after the search dates may provide valuable insights into current thinking.

To our knowledge, this is the first umbrella review to address the concerns of LLMs in health care research and practice. Thematic analyses provided insight into the complexity of different perspectives, and by using a whole population approach, it demonstrates common narratives. However, the poor quality of the included studies is a substantial limitation, and results should be interpreted with caution. Data quality is at the heart of these concerns, and combative action must ensure health care professionals and researchers have the resources required to overcome these apprehensions if AI is to be used routinely. Ethical, legal, and societal implications of AI use were also commonly raised. As technology accelerates and demands on health care increase, we must adapt and embrace change with equity, diversity, inclusion, and safety at the core.