A systematic review is the result of a systematic and rigorous evaluation of evidence, which may or may not include a meta-analysis [1]. Owing to the strict methodology and comprehensive summary of evidence, high-quality systematic reviews are considered the highest level of evidence, positioned at the top of the evidence pyramid [2]. Additionally, high-quality systematic reviews and meta-analyses are often used to support the development of clinical practice guidelines, aid clinical decision-making, and inform health care policy formulation [3]. Currently, the methods of systematic reviews and meta-analyses are applied in various disciplines in medicine and beyond such as law [4], management [5], and economics [6], and have yielded positive results, contributing to the continuous advancement of these fields [7].

The process of conducting systematic reviews demands a substantial investment in terms of time, resources, human effort, and financial capital [8]. To expedite the development of systematic reviews and meta-analyses, various automated or semiautomated tools such as Covidence have been developed [9,10]. However, the emergence of large language models (LLMs), particularly chatbots such as GPT, presents a set of both challenges and opportunities in the realm of systematic reviews and meta-analyses [11]. Based on the emerging literature in this field, we here provide our perspectives on the potential for harnessing the capabilities of LLMs to accelerate the production of systematic reviews and meta-analyses, while also scrutinizing the potential impacts and delineating the crucial steps involved in this process.

The procedures and workflows for conducting systematic reviews and meta-analyses are well-established. Currently, researchers often refer to the Cochrane Handbooks recommended by the Cochrane Library for intervention or diagnostic reviews [12,13]. In addition, some scholars and institutions have developed detailed guidelines on the steps and methodology for performing systematic reviews and meta-analyses [14-17]. Generally speaking, researchers should take the following steps to produce a high-quality systematic review and meta-analysis: determine the clinical question, register and draft a protocol, set inclusion and exclusion criteria, develop and implement a search strategy, screen the literature, extract data from included studies, assess the quality and risk of bias of included studies, analyze and processed data, write up the full text of the manuscript, and submit the manuscript for publication, as illustrated in Figure 1. These different steps contain many subtasks; therefore, conducting a complete systematic review and meta-analysis requires fairly complex and time-consuming work.

Although systematic reviews and meta-analyses have been widely applied and play an important role in developing guidelines and informing clinical decision-making, their production process faces many challenges. One of these challenges is the long production time and large resource requirements. The average estimated time to complete and publish a systematic review is 67.3 weeks, requiring 5 researchers and costing approximately US $140,000 [18,19]. More recently, the development of automated and semiautomated tools using natural language processing and machine learning have accelerated systematic review and meta-analysis production to some extent [20], with studies showing that such tools can help to produce a systematic review and meta-analysis within 2 weeks [21]. However, these tools also have some limitations. First, no single tool can fully accelerate the entire production process of systematic reviews and meta-analyses. Second, these tools cannot process and analyze literature written in different languages. Finally, the reliability of the results generated by these automated and semiautomated tools needs further validation as they are not yet widely adopted for this purpose.

Chatbots based on LLMs such as ChatGPT, Google Gemini, and Claude have become widely applied in medical research. These chatbots have proven to be valuable in tasks ranging from knowledge retrieval, language refinement, content generation, and medical exam preparation to literature assessment. ChatGPT has been shown to excel in accuracy, completeness, nuance, and speed when generating responses to clinical inquiries in psychiatry [22]. Moreover, LLMs such as ChatGPT play a pivotal role in automating the evaluation of medical literature, facilitating the identification of accurately reported research findings [23]. Despite their significant contributions, these chatbots are not without limitations. Challenges such as the potential for generating misleading content and susceptibility to academic deception necessitate further scholarly discourse on effective mitigation strategies. Standardized reporting practices may contribute to delineating the applications of ChatGPT and mitigating research biases [24].

ChatGPT has also demonstrated significant application potential and promise in the process of conducting systematic reviews and meta-analyses. Various studies [11,25-32] indicate that ChatGPT can play a pivotal role in formulating clinical questions, determining inclusion and exclusion criteria, screening literature, assessing publications, generating meta-analysis code, and assisting the full-text composition, among other relevant tasks. The details of these capabilities are summarized in Table 1.

Systematic reviews and meta-analyses are crucial evidence types that support the development of guidelines [3]. The benefits of employing LLM-based chatbots in the production of systematic reviews and meta-analyses include increased speed, such as in the stages of evidence searching, data extraction, and assessment of bias risk; these tools can also enhance accuracy by reducing human errors such as those made while extracting essential information and pooling data. However, there are also drawbacks of these applications of LLMs, such as the potential for generating hallucinations, the requirement for human verification owing to the poor reliability of the models, and that the entire systematic review process is not replicable. Moreover, when interacting with LLM chatbots, it is important to manage data privacy. In particular, when using LLMs to analyze data, especially when including personal patient information, ethical approval and management must be properly addressed.

While LLMs can assist in accelerating the production of systematic reviews and meta-analyses in some steps, enhancing accuracy and transparency, and saving resources, they also face several challenges. For instance, LLMs cannot promptly update their versions and information. For example, ChatGPT 3.5 has been trained on data available in 2021. Thus, limitations such as the length of prompts and token constraints, as well as restrictions related to context associations, may potentially impact the overall results and user experience [25]. Although LLM-based autonomous agents have made strides in tasks related to systematic reviews and meta-analyses, their applications are still associated with various issues related to personalization, updating knowledge, strategic planning, and complex problem-solving. The development of LLM-driven autonomous agents adept at systematic reviews and meta-analyses warrants further exploration [66]. The use of LLMs as centrally controlled intelligent agents encompasses the ability to handle precise literature screening, extract and analyze complex data, and assist in manuscript composition, as highlighted by proof-of-concept demonstrations such as MetaGPT [67]. Moreover, the continuous growth of the use of LLMs can pose a significant challenge in ensuring the accuracy of information provided in systematic reviews, particularly if LLMs are indiscriminately overused.

To better facilitate the use of tools such as ChatGPT in systematic reviews and meta-analyses, we believe that, first and foremost, authors should understand the scope and scenarios for applying ChatGPT, clearly defining which steps can benefit from these tools. Second, for researchers, collaboration with computer scientists and AI engineers is crucial to optimize the prompts and develop integrated tools based on LLMs, such as web applications. These tools can assist in seamless transitions between different tasks in the systematic review process. Lastly, for journal editors, collaboration with authors and reviewers is essential to adhere to reporting and ethical principles associated with the use of GPT and similar tools [24,68]. This collaboration aims to promote transparency and integrity, while preventing indiscriminate overuse in the application of LLMs in systematic reviews and meta-analyses.

The emergence of LLMs could have a significant impact on the production of systematic reviews and meta-analyses. In this process, the application of chatbots such as ChatGPT has the potential to speed up certain steps such as literature screening, data extraction, and risk of bias assessment, which are processes that typically consume a considerable amount of time. However, it is important to note that if AI methods such as GPT are employed in performing systematic reviews, disclosure and declaration of the use of these tools are essential. This includes specifying the AI tools used, their roles, and the areas of application within the review process, among other relevant information for full disclosure [24]. In this context, developing a reporting guideline is warranted to guide the application of LLM tools in systematic reviews and meta-analyses. Although the PRISMA 2020 guideline briefly addresses the use of automation technologies, its coverage is limited to steps such as screening, and there is a lack of comprehensive guidance on the broader spectrum of applications [64].