Due to the nature of the study, institutional review board approval was not required, as the research did not involve patient data and did not constitute human participants.

In this study, we focused on case reports highlighting instances of misdiagnoses resulting from cognitive biases. Our research involved a comprehensive search of the PubMed database using the terms “case reports [Publication Type]” AND “cognitive bias[All Fields]”. PubMed was chosen for its comprehensive coverage of case studies across diverse medical disciplines, ensuring access to a broad spectrum of peer-reviewed case studies for our analysis.

Eligibility for inclusion requires case reports to meet four key criteria: (1) they must provide detailed case information sufficient for making the initial diagnosis; (2) they must include a final, accurate diagnosis for the patient; (3) the incorrect diagnosis must be linked to cognitive bias by the authors; and (4) the final diagnosis should not be a rare disease or unclear. A rare disease is a disease or condition that affects fewer than 200,000 patients per year. The list of exclusions for rare diseases was based on the National Organization for Rare Disorders [28]. We set no limits on the publication year of these reports.

Screening of abstracts for eligible studies was conducted by 2 independent clinically trained reviewers, YK and TXYL. Each reviewer separately assessed whether a case should be included or excluded based on predefined criteria. In instances of disagreement, SAL reviewed the justifications for exclusion and inclusion and made the final decision. The full texts were reviewed to obtain the case summary, the initial wrong diagnosis by the medical team, and the final diagnosis of the case reported.

For the studies included in the analysis, full-text extraction, including patient demographics, past medical history, initial presenting complaints, and results of the preliminary investigations, was conducted. For cases involving imaging data, such as chest x-rays, we did not incorporate the actual images into the query. Instead, we opted to include the legends or descriptions accompanying these images. In defining the boundaries of the clinical scenarios for our study, we restricted the information to that available up to the point of and before the initial diagnosis. This meant deliberately excluding any subsequent investigations, treatments, or management strategies that followed.

As GPT-4 Turbo has a knowledge base trained up to April 2023 [29], there is potential bias stemming from the inclusion of case reports that might have been part of the LLM’s pre-training data. Hence a personal clinical scenario that was not published on the internet was included. This complex case was derived from the critical care attending’s personal experience where cognitive biases had resulted in wrong and delayed diagnosis. A concise summary of these clinical scenarios, including the unique case, is provided in Multimedia Appendix 1.

In this study, we used the multi-agent conversation framework provided by AutoGen [22] to assess its efficacy in mitigating cognitive biases in clinical decision-making. Within the system, each agent interacts based on their predefined role prompts, thereby simulating the collaborative decision-making process typically observed among health care professionals.

The suggested optimal group size to facilitate group discussion and performance has been proposed to be between 3 and 5 [30]. In the absence of established literature recommending an optimal team size for mitigating cognitive biases in medical settings, we constructed a simulation using 3 to 4 different agents, representing a typical clinical team composition [31]. These configurations aim to realistically emulate the dynamics of clinical teams and their potential to reduce cognitive biases.

As shown in Table 1, we tested 3 different frameworks to identify the most effective configuration. Framework 3 consisted of 3 agents (Junior Resident I, Junior Resident II, and Recorder), while Frameworks 4 (Junior Resident I, Junior Resident II, Professional Expert, and Recorder) and 4-C (Junior Resident I, Junior Resident II, Senior Doctor, and Recorder) each utilized 4 agents, with the fourth agent playing different roles. The distinguishing feature of Framework 4-C, in contrast to other frameworks, lies in its explicit directive for the Senior Doctor role to engage in discussions specifically focused on cognitive biases alongside the initial diagnosis. In addition, we experimented with combinations involving 5 or more agents, but the fifth agent did not effectively participate in the conversations despite modifications to the prompts. Consequently, the frameworks were limited to a maximum of 4 agents. All prompts for agent roles can be found in Multimedia Appendix 2.

The diagnostic process was orchestrated through the collaborative efforts of simulated medical professionals (agents) with varying levels of expertise, as shown in Figure 1. Junior Resident I, as the primary physician, was tasked with presenting the initial diagnosis. Junior Resident I was given the personality of making swift assumptions but is willing to embrace feedback and consider alternative diagnostic possibilities. After the group discussion, Junior Resident I is then allowed to reconsider the most probable differential diagnosis along with an alternative. Junior Resident II, a colleague of Junior Resident I, critically appraised the initial diagnosis, pinpointing inconsistencies and advocating for alternative differential diagnoses. This role was instrumental in addressing potential confirmation and anchoring biases in the diagnostic process. Complementing the juniors, the Senior Doctor brought in-depth experience to the table, crucially identifying cognitive biases in the initial diagnosis and steering the junior residents toward a more nuanced and accurate diagnosis, while the Professional Expert aims to provide any specialist knowledge required to help with the diagnosis without further encouraging discussions of cognitive biases. This guidance was vital in circumventing premature diagnostic closure and knowledge bias. In addition, the role of the Recorder was to distill the outcomes of the discussion, compiling a definitive list of differential diagnoses and extracting key learning points from the collaborative effort, thereby enriching the diagnostic process with a comprehensive and multifaceted approach.

The final accurate diagnoses for the clinical scenarios were extracted directly from the published cases. Summarized answers from both the multi-agent framework and human evaluators were marked as “Correct” if they matched the final accurate diagnoses. Vague answers, such as diagnosing “septic shock” when the accurate diagnosis was “endometriosis,” were marked as “Incorrect.” A total of 2 physicians graded the answers. In cases where there were discrepancies in their assessments, discussions were held to reach a consensus.

The overall performance of the framework was evaluated based on the accuracy of (1) the “initial diagnosis” made without any multi-agent discussions and (2) the “final diagnosis” after the discussions. Each clinical scenario within each multi-agent framework was simulated 5 times to assess the consistency of diagnoses across multiple iterations.

A total of 3 doctors, each with at least 5 years of clinical experience, were asked to provide their top differential diagnosis along with 2 other differentials based on the clinical scenarios. If the top differential was correct, both the “initial diagnosis” and “final diagnosis” were marked as correct. The diagnoses generated by the multi-agent framework were compared to those provided by human doctors using Fisher exact test.

An integral part of the evaluation involved documenting the specific cognitive biases identified and addressed during the agents’ discussions. This aspect focused on understanding how effectively the multi-agent system could recognize and mitigate cognitive biases, which are crucial factors in diagnostic accuracy. Hallucinations are characterized by the dissemination of false medical information during multi-agent conversations or responses that fail to directly address the queries posed. This determination was made following an independent review by 2 doctors who thoroughly evaluated the provided answers. The interaction and decision-making process among the agents are illustrated in (Figure 2). This representation aids in visualizing the dynamics of the simulation and the interplay between different agents in reaching a diagnosis.

A comprehensive search of the PubMed database yielded 162 case reports, of which 37 were determined to be eligible for inclusion in the study. From this subset, 15 cases were selected for evaluation as a representative sampling. In addition, a 16th scenario, derived from critical care attending personal clinical experience, was included. The flow diagram can be viewed in Figure 3.

A total of 240 responses were generated by the multi-agent conversation framework, encompassing both initial and final diagnoses following discussions with the agents. The initial diagnosis made by the first-responder agent had a correctness rate of 0% (0/80) across all 3 multi-agent frameworks, whereas the human-generated answers had an accuracy rate of 27% (13/48). After the multi-agent discussions, the final diagnosis was correct in 76% (61/80) of cases in Framework 4-C, which was significantly better than the human answers (odds ratio 3.49; P=.002), as shown in Table 2.

The clinical cases covered a broad spectrum of medical fields, ranging from pediatric to malignancy diagnosis. Specifically, 6 cases were centered on infectious disease diagnosis, 3 pertained to critical care, and 2 involved vascular-related diagnoses. The rest were diverse, spanning neurology, gynecology, cancer, urology, and endocrinology (Table 3). A notable aspect of these cases was the cross-disciplinary nature of most initial and final diagnoses, observed in 12 (75%) out of the 16 cases. For example, in one illustrative case (case 1 [32]), an older adult patient presented with symptoms of shortness of breath on exertion and cough, leading to an initial misdiagnosis of heart failure. However, further investigation, considering her ongoing treatment with infliximab for rheumatoid arthritis, revealed the actual diagnosis of miliary tuberculosis. Another case (case 12 [33]) involved a young woman presenting with sudden, left-sided sharp pleuritic chest pain, which lessened when sitting forward. Despite the initial chest radiograph being interpreted as showing no acute abnormalities, the AutoGen system, provided with this potentially misleading information, initially diagnosed the condition as a pulmonary embolism. Yet, after a thorough discussion and re-evaluation, the correct diagnosis of pneumothorax was established, indicating a missed finding in the chest radiograph.

There were variations observed in the repeated scenarios, particularly in the process of generating the top 2 differential diagnoses. For instance, in case 13 [44], a young patient presenting with lactic acidosis was initially diagnosed with diabetic ketoacidosis, and further discussions within the multi-agent environment led to the identification of thiamine deficiency. In 3 (60%) out of 5 simulations, Junior Resident I identified thiamine deficiency as the top differential diagnosis following the multi-agent discussions. In the remaining 2 simulations, gastrointestinal disorders were initially considered the most likely diagnosis, with thiamine deficiency being the second most likely differential.

The multi-agent discussions led to the correct final diagnosis in 13 (81%) out of the 16 scenarios across all 3 multi-agent frameworks. Furthermore, these discussions were effective in identifying various clinical biases, including anchoring bias, confirmation bias, availability bias, and premature closure. A detailed breakdown of the answers and the cognitive biases identified is provided in Multimedia Appendix 1. The detailed breakdown of correct answers for each scenario is available in Multimedia Appendix 3.

This study assessed the effectiveness of the multi-agent conversation framework in improving diagnostic accuracy and mitigating cognitive biases in clinical decision-making. Our findings reveal that integrating multi-agent discussions substantially enhances diagnostic accuracy. The best-performing framework, which used 4 agents, included 1 agent specifically tasked with identifying cognitive biases. This 4-C multi-agent framework that includes discussions on cognitive biases performed significantly better compared with human-generated answers. However, it is important to note that while 4 agents performed best in our study setting, this may vary in general applications.

The increase in diagnostic accuracy demonstrates the value of multi-perspective analysis in medical diagnosis, a core feature of the multi-agent conversation environment. This is in line with previous research emphasizing the importance of collaborative decision-making in health care to mitigate individual cognitive biases [47]. The consistency of responses in the repeated scenarios further validates its reliability and potential applicability in real-world clinical settings.

The effectiveness of the multi-agent conversation system, particularly in scenarios involving misleading or misinterpreted initial investigations, is noteworthy. This was exemplified in case 12, where our multi-agent framework successfully identified a pneumothorax that had been initially overlooked by human clinicians, and improved the accuracy of the final diagnosis to 100% (5/5) in multi-agent framework 4-C. The multi-agent systems were able to critically examine and question potential misinterpretations. Such capabilities are crucial in refining the diagnostic process and enhancing accuracy. While Brown et al [33] discussed the role of AI in the identification of pneumothorax, there is a potential for pre-trained LLMs to replace the decision aid, rather than to develop new resource-intensive systems such as image deep learning. This strategy aligns with the current trajectory of AI development in health care, where the focus is on integrating and maximizing existing AI technologies to enhance clinical decision-making and focus on sustainable AI [48,49].

The integration of multi-agent frameworks in clinical practice holds promising implications for enhancing diagnostic accuracy and ultimately improving patient outcomes. By systematically addressing cognitive biases through collaborative discussions among agents, these frameworks offer a structured approach to refining diagnostic reasoning. This approach not only complements traditional diagnostic methods but also introduces a dynamic element that challenges and verifies initial clinical hypotheses. In practical terms, the ability of multi-agent systems to consistently identify and correct potential diagnostic errors, as demonstrated in our study, suggests a transformative potential in reducing patient morbidity and optimizing treatment strategies. Furthermore, the application of these frameworks within electronic medical records (EMRs) could revolutionize decision-making processes by providing real-time, data-driven insights that augment clinician judgment and ensure adherence to best practices. As health care systems evolve toward more integrated and technology-driven approaches, the strategic incorporation of multi-agent systems stands poised to contribute significantly to improving the quality and efficiency of patient care delivery.

The results of our study extend beyond the educational benefits of multi-agents, highlighting their potential for broader clinical integration. The reflective process fostered by engaging with LLMs in diagnosing and revising cases not only cultivates an educational atmosphere conducive to developing critical thinking skills but also suggests practical applications in clinical settings [50]. One notable avenue is the integration of multi-agent into EMR systems. This could enhance decision-making processes by providing real-time, data-driven insights and augmenting the cognitive capabilities of medical professionals. Such integration would not only streamline the diagnostic process but also aid in the identification of potential cognitive biases, thereby enhancing the quality of patient care. Furthermore, the incorporation of multi-agents in EMRs could facilitate continuous learning and improvement, ensuring that medical practitioners remain updated with the latest medical knowledge and best practices, crucial for maintaining high standards in patient treatment and care.

The study, while providing valuable insights into the potential application of multi-agents in clinical diagnostics, is subject to several limitations. First, the reliance on published case reports limits the breadth of clinical scenarios, potentially affecting the generalizability of our findings to broader medical practice. Second, the exclusion of visual data, such as medical imaging, confines our model’s diagnostic capabilities to text-based information, omitting a critical component of clinical diagnosis. In addition, the inherent biases present in the LLMs, based on their pre-training data, could have influenced the diagnostic suggestions. Meanwhile, the technical limitations inherent in LLMs, including their understanding of complex medical terminologies and nuances [51], may not match the expertise of experienced clinicians, possibly limiting the scope of applicability.

Future studies could assess the effectiveness and adaptability of the multi-agent framework in evolving clinical scenarios. More importantly, while LLMs have demonstrated potential as a valuable clinical aid in correcting cognitive biases, the implementation of such technology in health care necessitates rigorous ethical and regulatory oversight [52] and should continue to augment rather than replace the human clinician’s expertise [14].

In conclusion, our study highlights the potential of multi-agents in enhancing diagnostic accuracy in clinical decision-making. The findings support the integration of advanced generative AI technology in educational and clinical settings as a tool for augmenting human decision-making. Future research should focus on the application of these systems in real-time clinical environments and their impact on patient outcomes. Moreover, ethical and legal considerations regarding the use of AI in health care need continued exploration to ensure patient safety and professional accountability.