This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
Background
The COVID-19 pandemic, caused by the SARS-CoV-2 virus, has necessitated reliable and authoritative information for public guidance. The World Health Organization (WHO) has been a primary source of such information, disseminating it through a question and answer format on its official website. Concurrently, ChatGPT 3.5 and 4.0, a deep learning-based natural language generation system, has shown potential in generating diverse text types based on user input.
Objective
This study evaluates the accuracy of COVID-19 information generated by ChatGPT 3.5 and 4.0, assessing its potential as a supplementary public information source during the pandemic.
Methods
We extracted 487 COVID-19–related questions from the WHO’s official website and used ChatGPT 3.5 and 4.0 to generate corresponding answers. These generated answers were then compared against the official WHO responses for evaluation. Two clinical experts scored the generated answers on a scale of 0-5 across 4 dimensions—accuracy, comprehensiveness, relevance, and clarity—with higher scores indicating better performance in each dimension. The WHO responses served as the reference for this assessment. Additionally, we used the BERT (Bidirectional Encoder Representations from Transformers) model to generate similarity scores (0-1) between the generated and official answers, providing a dual validation mechanism.
Results
The mean (SD) scores for ChatGPT 3.5–generated answers were 3.47 (0.725) for accuracy, 3.89 (0.719) for comprehensiveness, 4.09 (0.787) for relevance, and 3.49 (0.809) for clarity. For ChatGPT 4.0, the mean (SD) scores were 4.15 (0.780), 4.47 (0.641), 4.56 (0.600), and 4.09 (0.698), respectively. All differences were statistically significant (P<.001), with ChatGPT 4.0 outperforming ChatGPT 3.5. The BERT model verification showed mean (SD) similarity scores of 0.83 (0.07) for ChatGPT 3.5 and 0.85 (0.07) for ChatGPT 4.0 compared with the official WHO answers.
Conclusions
ChatGPT 3.5 and 4.0 can generate accurate and relevant COVID-19 information to a certain extent. However, compared with official WHO responses, gaps and deficiencies exist. Thus, users of ChatGPT 3.5 and 4.0 should also reference other reliable information sources to mitigate potential misinformation risks. Notably, ChatGPT 4.0 outperformed ChatGPT 3.5 across all evaluated dimensions, a finding corroborated by BERT model validation.
The COVID-19 pandemic, caused by the SARS-CoV-2, has had a profound global impact [1]. As of June 1, 2023, the pandemic has resulted in over 767 million reported cases and over 6.938 million fatalities worldwide, marking it as one of the most significant pandemics in human history [2]. The complex transmission modes, extended incubation period, atypical symptoms, and emergence of multiple variants pose substantial challenges for pandemic prevention, control, and treatment [3].
Efforts to prevent and treat COVID-19 continue unabated, and there is a high public demand for related information [4,5]. World Health Organization (WHO) [2], a leading authority in public health, has published a series of frequently asked questions about COVID-19 on its official website [6]. These frequently asked questions provide comprehensive coverage on various aspects of COVID-19, including basic knowledge, transmission modes, prevention methods, treatments, and its impact on different populations and environments [7-9]. However, the sheer volume of information, frequent updates, and potential language barriers may hinder access and comprehension, leading to misinformation [5,10,11].
The advent of artificial intelligence (AI) technology has seen the rise of dialog models that are gradually replacing traditional search engines. These models, based on large language models, use deep learning to generate natural language text in various formats, such as questions and answers (Q&As), summaries, and stories, based on user input [12]. ChatGPT, an advanced dialog model, leverages a large corpus and powerful neural networks to generate fluent, coherent, and logical text. It has found applications in numerous fields, including medical information provision, education, and scientific research, offering users convenient and efficient information services [13-16].
This study aims to assess ChatGPT’s capability as a COVID-19 information service platform, providing the public with accurate and relevant information about the virus [17,18]. This research not only evaluates the performance of ChatGPT in disseminating COVID-19 information but also offers insights into other informational services related to epidemics.
MethodsEthical Considerations
This study was conducted in alignment with the Declaration of Helsinki and did not necessitate ethics committee approval.
Study Design
The research was executed in 2 stages. In the initial stage, we extracted 487 questions related to COVID-19 from the WHO official website and used ChatGPT 3.5 and 4.0 to generate corresponding answers (Multimedia Appendix 1). Two clinicians were invited to score these generated answers, referencing the authoritative WHO responses. The scoring evaluated the quality of the answers across 4 dimensions: accuracy, comprehensiveness, relevance, and clarity. Each answer was assigned a score from 0 to 5 based on a predefined scoring standard (Table 1). Concurrently, we used the BERT (Bidirectional Encoder Representations from Transformers) model to compute the similarity score between the generated answers and the official WHO responses, with scores ranging from 0 (completely dissimilar) to 1 (identical).
The scoring system (0-5) used for evaluating COVID-19 information from ChatGPT 3.5 and 4.0.
Criteria
0 points
1 points
2 points
3 points
4 points
5 points
Accuracya
Completely wrong or irrelevant
Mostly wrong or irrelevant
Partially wrong or irrelevant
Few wrong or irrelevant
Mostly correct or relevant
Completely correct or relevant
Completenessb
Completely missing or redundant
Mostly missing or redundant
Partially missing or redundant
Few missing or redundant
Mostly covered or concise
Completely covered or concise
Relevancec
Completely deviated or unrelated
Mostly deviated or unrelated
Partially deviated or unrelated
Few deviated or unrelated
Mostly close or related
Completely close or related
Clarityd
Completely vague or ambiguous
Mostly vague or ambiguous
Partially vague or ambiguous
Few vague or ambiguous
Mostly clear or explicit
Completely clear or explicit
aAccuracy: measures the factual correctness.
bComprehensiveness: evaluates the breadth or depth of information.
cRelevance: assesses how directly the information relates to COVID-19.
dClarity: scores readability and understandability.
In the second stage, we conducted a quantitative and qualitative analysis of the first-stage data, comparing it with the official WHO information to assess the quality of the COVID-19 information generated by ChatGPT 3.5 and 4.0. This analysis facilitated a discussion on the strengths and limitations of the answers generated by ChatGPT 3.5 and 4.0 and allowed us to propose suggestions for improvement.
Data Source
All questions and answers used in this study were sourced from the Q&A section about COVID-19 on the official WHO website. This website is a primary source of authoritative and reliable COVID-19 information, with its content undergoing professional and scientific review and updates. We extracted 487 questions covering various aspects of COVID-19, such as basic knowledge, transmission routes, preventive measures, vaccination, and travel advice, as samples for this study. These questions were input into ChatGPT 3.5 and 4.0 to generate corresponding answers, which were then compared with the official WHO responses to form the data set for this study. To mitigate the influence and bias of context association in information generation, we used 2 separate accounts, with each question being asked in a newly created dialog box. The complete list of prompts used for this purpose with ChatGPT 3.5 and 4.0 can be found in Multimedia Appendix 2.
Data Processing and Analysis MethodsExpert Scoring
Data processing and statistical analysis of clinicians’ evaluations were executed using RStudio software (version 1.1.35; PBC). Two clinicians, hailing from tier-3 class-A hospitals in China and with substantial contributions to China’s COVID-19 response, independently scored the answers generated by ChatGPT 3.5 and ChatGPT 4.0. Scoring was carried out across 4 predetermined dimensions—accuracy, comprehensiveness, relevance, and clarity—and was benchmarked against the official answers provided by the WHO. Both clinicians were blinded to the source of the answers, ensuring a double-blind evaluation process. Additionally, the sequence of answers for each question was randomized to further minimize bias. Prior to the evaluation, the clinicians consulted an authoritative compendium of COVID-19 questions and answers from the WHO to ensure a comprehensive and accurate understanding of the subject matter. The individual clinical evaluation scores by KG are detailed in Multimedia Appendix 3, and the scores by QL can be found in Multimedia Appendix 4.
We examined the consistency of the scores from the 2 clinicians, calculating the Cronbach α coefficient of the scores for both versions. Furthermore, we performed a descriptive statistical analysis of the average scores of the generated answers across the 4 dimensions and compared them with the official WHO answers. Before conducting hypothesis testing, the distribution of the data across the 4 dimensions, accuracy, comprehensiveness, relevance, and clarity, was considered for both versions of ChatGPT. Given that the Mann-Whitney U test does not assume normality of the data distribution, this nonparametric test was directly applied to evaluate the statistically significant differences between the responses generated by ChatGPT 3.5 and 4.0, which is especially appropriate for our data as it does not require the assumption of normality.
BERT Scoring
In this study, the BERT model, a pretrained deep learning model renowned for its efficacy in natural language processing tasks, was used to appraise the quality of responses generated by ChatGPT 3.5 and ChatGPT 4.0. The BERT model is adept at identifying intricate semantic patterns in text, thereby generating high-quality text representations [19]. We calculated the cosine similarity between the vector representations of the authoritative responses from the WHO and the responses generated by ChatGPT 3.5 and ChatGPT 4.0. The closer the calculated value is to 1, the higher the semantic congruence between the generated response and the authoritative answer. This method provides a quantitative measure of the quality of the information provided by the AI models in relation to the authoritative source. A detailed comparison of the BERT scores and the responses is presented in Multimedia Appendix 5.
ResultsExpert Scoring
Using the Mann-Whitney U test, we discerned statistically significant disparities across all assessed dimensions, namely, accuracy, comprehensiveness, relevance, and clarity (each with P<.001). Notably, ChatGPT 4.0 outperformed ChatGPT 3.5 in every evaluated dimension, corroborating the hypothesis that ChatGPT 4.0 is superior in generating responses that are not only accurate but also comprehensive, relevant, and clear (Table 2).
Statistical comparison of ChatGPT 3.5 and ChatGPT 4.0 across evaluation dimensions.
Evaluation dimension
Score for ChatGPT 3.5, mean (SD)
Score for ChatGPT 4.0, mean (SD)
Mann-Whitney U value
P value
Accuracy
3.47 (0.725)
4.15 (0.780)
263,250
<.001
Comprehensiveness
3.89 (0.719)
4.47 (0.641)
283,632
<.001
Relevance
4.09 (0.787)
4.56 (0.600)
328,018
<.001
Clarity
3.49 (0.809)
4.09 (0.698)
294,482
<.001
The consistency of the scores assigned by the 2 experts to the responses generated by both versions of ChatGPT was rigorously evaluated. This evaluation was grounded on the detailed scoring provided in Multimedia Appendices 3 and 4. The Cronbach α coefficients for the scores from ChatGPT 3.5 and 4.0 were .94 and .92, respectively, indicating a high degree of consistency in the evaluations made by the 2 experts. These coefficients, significantly exceeding .9, denote a robust agreement between the experts in their assessment methods. This level of interrater reliability not only confirms the consistency of the expert evaluations but also enhances the validity of our study's conclusions. The values provided above are directly derived and calculated from the detailed scores found in Multimedia Appendices 3 and 4. The high α values, approaching 1, signify a strong consensus in the expert evaluations, reinforcing the reliability and credibility of their assessments of the answers generated by the 2 different versions of ChatGPT.
BERT Scoring
The average similarity scores between the responses generated by ChatGPT versions 3.5 and 4.0 and the official WHO responses are discussed here. Both versions achieved similarity scores above 0.8. Specifically, ChatGPT 4.0 scored slightly higher with an average BERT score of 0.85 (SD 0.07) compared to ChatGPT 3.5, which scored an average of 0.83 (SD 0.07). This suggests that ChatGPT 4.0 has made improvements in terms of semantic similarity. For a detailed view of the responses generated by both ChatGPT 3.5 and ChatGPT 4.0 for the COVID-19 Q&A, refer to Multimedia Appendix 6.
Descriptive Analysis
Our analysis revealed that the responses generated by ChatGPT 3.5 and 4.0 to certain questions were on par with the authoritative responses from the WHO, as demonstrated by high clinical expert ratings and BERT scores (Figures 1-3).
World Health Organization answer—“What should I do if I have COVID-19 symptoms?”.
ChatGPT 3.5 answer—“What should I do if I have COVID-19 symptoms?”.
ChatGPT 4.0 answer—“What should I do if I have COVID-19 symptoms?”.
However, we also identified areas where ChatGPT struggled to provide accurate responses. For instance, it was unable to provide information on the Omicron variant, as this is the knowledge that emerged after September 2021, beyond its training data (Figures 4-6). Furthermore, ChatGPT 4.0 performed poorly on topics related to humanities and ethics. For example, it was unable to provide effective assistance in the scenario of women facing domestic violence during the COVID-19 pandemic (Figures 7-9).
World Health Organization answer—“What is the Omicron variant?”.
ChatGPT 3.5 answer—“What is the Omicron variant?”.
ChatGPT 4.0 answer—“What is the Omicron variant?”.
World Health Organization answer—“I have harmed or am worried about harming or hurting my partner (and children) with my words or actions. How can I stop?”.
ChatGPT 3.5 answer—“I have harmed or am worried about harming or hurting my partner (and children) with my words or actions. How can I stop?”.
ChatGPT 4.0 answer—“I have harmed or am worried about harming or hurting my partner (and children) with my words or actions. How can I stop?”.
A visual comparison of the key points derived from the responses of the WHO, ChatGPT 3.5, and ChatGPT 4.0 to a specific question is provided in Figure 10. This comparison demonstrates the ability of ChatGPT 3.5 and ChatGPT 4.0 to generate reliable and accurate responses, with ChatGPT 4.0 offering more comprehensive and nuanced perspectives.
Comparing key points in ChatGPT 3.5 and 4.0 and WHO responses to question 39. WHO: World Health Organization.
DiscussionPrincipal Findings
Through the evaluation of COVID-19 information generated by ChatGPT and authoritative information from the WHO, we find that the advantages of ChatGPT in generating COVID-19 information lie in its ability to generate comprehensive and relevant information, but there is still room for improvement in the accuracy and clarity of the information generated [20]. Although the information generated by ChatGPT 4.0 is superior to ChatGPT 3.5 in terms of accuracy, comprehensiveness, relevance, and clarity, there are still limitations; especially, when facing complex ethical situations, it cannot provide specific and effective suggestions. This is significant for understanding and improving the performance of ChatGPT, enhancing its application value in the field of public health, and promoting the cooperation between AI technology and public health institutions. This study also provides reference and inspiration for other epidemic information services, demonstrating the potential and challenges of generative dialog models in handling complex and sensitive information [21-23].
Comparison to Prior Work
This study is the first to include a complete authoritative official Q&A database on COVID-19 for comparison, in order to assess the quality of the COVID-19 information generated by ChatGPT. A research design combining quantitative and qualitative methods was adopted, and a comprehensive and in-depth analysis of the generated answers was conducted from multiple dimensions of expert scoring and BERT similarity scoring. The performance differences between ChatGPT 3.5 and ChatGPT 4.0 are compared to reflect the evolution speed and direction of the ChatGPT model. This is a dynamic and comparative study that provides a benchmark or reference point for other versions of ChatGPT [24].
Future Directions
We found that ChatGPT performs excellently in many areas, but it also has the following limitations [25-27]. First, it is time-limited, as it only contains information up until September 2021. Therefore, it cannot explain or answer some new concepts, such as the Omicron variant, which was first reported to the WHO on November 24, 2021, and listed as a variant of concern by the WHO on November 26, 2021. Although ChatGPT 4.0 cannot accurately describe the Omicron variant, it can enumerate all known variants and describe possible mutations, making its answers more comprehensive and relevant than those of ChatGPT 3.5. Second, it does not annotate its sources, which makes immediate verification difficult. Almost all answers given by ChatGPT do not annotate their sources, making it hard to verify the authenticity of data and information. However, in general, the accuracy of answers from ChatGPT 4.0 is higher than that of ChatGPT 3.5. Third, its responses to professional information are somewhat vague, such as those related to the treatment of COVID-19. It can accurately list the types and schemes of drugs used in COVID-19 treatment, but neither ChatGPT 4.0 nor ChatGPT 3.5 can provide standard protocols for drug use and dosage. Therefore, ChatGPT is more suitable for assisting medical workers rather than replacing them. Fourth, it may struggle to handle questions related to ethics [28]. When we asked questions related to ethics, the answers were often vague. For example, ChatGPT 4.0 may suggest that we seek help from a trusted person, but this answer is neither accurate nor comprehensive, and it does not solve the actual problem. We look forward to new versions of ChatGPT that have real-time training data and make greater progress in areas such as information citation, professionalism, and ethics.
Building on the limitations discussed, it is crucial to consider the ethical dimensions that come with the application of AI in public health. These concerns are not merely theoretical but have practical implications for the integrity of health care services and public trust. In addressing these ethical concerns, we emphasize the importance of safeguarding data privacy through robust protections, mitigating misinformation with stringent validation of AI-generated content, and enhancing the ethical reasoning capabilities of AI systems [29,30]. As AI’s role in health care grows, it is imperative that these systems not only provide accurate information but also align with ethical standards to support the integrity of health care delivery.
Compared to traditional search engines, ChatGPT can provide continuous, customized, multichannel, and user-friendly information services. It can help the public obtain and understand authoritative and accurate information in the field of public health, thereby improving their health awareness and behavior, reducing their risk of infection or spread of diseases, relieving their psychological pressure and anxiety, and enhancing their confidence and optimistic attitude [31]. In this study, ChatGPT 4.0 outperformed ChatGPT 3.5 in terms of accuracy, comprehensiveness, relevance, semantic similarity, and information matching, indicating the continuous evolution and optimization of the ChatGPT model. The answers to the COVID-19-related questions from ChatGPT 4.0 have a high consistency with the official answers from WHO, with scores in 4 dimensions exceeding 4, indicating that ChatGPT 4.0 can serve as an effective and relatively reliable information service tool to help the public cope with the global pandemic of COVID-19. Of course, we also look forward to the updates of more advanced versions to improve the accuracy and clarity of generated questions and provide accurate answers to professional questions.
Strengths and Limitations
Despite the promising results, there are some limitations in this study. First, the evaluation was conducted by only 2 clinicians, whose assessments may be influenced by personal preferences and subjective judgments [32]. They may not fully understand and evaluate the answers generated by ChatGPT, thereby potentially limiting the reliability and validity of expert scoring. To address this, we used Cronbach α as a statistical measure of scoring consistency, which showed a high degree of agreement (α value greater than .9) between the evaluators, indicating minimal bias. Nonetheless, we recognize the value of a broader panel of evaluators. Future studies could benefit from a more diverse group of experts for further validation and will strive to include experts from various medical specialties and geographic locations. Additional statistical methods will also be considered to adjust for individual rater biases, thus enhancing the robustness of our research findings. Second, although our primary use of the BERT model as a scoring tool involves calculating similarity scores to assess the quality of responses compared to authoritative answers, we are aware that this method may not capture all subtle semantic differences [33]. Therefore, we also included expert evaluations as a complement, which are not limited by complex semantics and can assess the quality of responses from additional dimensions. The results consistently show that ChatGPT 4.0 outperforms ChatGPT 3.5 in expert assessments, addressing potential limitations of BERT scoring. Future research will explore the inclusion of a more diverse set of natural language processing models to further enhance our understanding and assessment of the semantic depth of AI-generated content. Finally, the study evaluated the quality of generated answers only from the perspectives of doctor scores and BERT scores, without considering subjective factors such as user satisfaction. This may not fully reflect users’ perception of the quality of generated answers. Although we obtained consistent conclusions in the 2 tests, we hope that more tests based on more epidemic information can help us verify the potential of ChatGPT in providing information on epidemics in the future [34].
Conclusions
In conclusion, this study offers a comparative analysis of the quality of COVID-19 information generated by ChatGPT 3.5 and ChatGPT 4.0, benchmarked against the authoritative information provided by the WHO. Our findings indicate that ChatGPT 4.0 has surpassed its predecessor, ChatGPT 3.5, in multiple dimensions and exhibits a higher degree of similarity to the WHO’s official information. This conclusion is further corroborated by our tests using the BERT model.
Nevertheless, there remains a significant gap between the accuracy and clarity of the responses generated by ChatGPT 4.0 and the WHO’s official information, indicating areas for potential enhancement. Conversely, in terms of comprehensiveness and relevance, the responses generated by ChatGPT 4.0 demonstrate commendable performance, occasionally even exceeding the WHO’s official information.
This research contributes to our understanding and potential improvement of ChatGPT’s performance, thereby enhancing its applicability in the realm of public health and fostering collaboration between large language models and public health organizations. As an innovative, systematic, in-depth, dynamic, and comparative study, our research provides valuable insights and serves as a reference for other epidemic information services and generative dialog models.
World Health Organization's question and answer collection on COVID-19.
Complete list of prompts used for ChatGPT 3.5 and ChatGPT 4.0 in COVID-19 question and answer evaluation.
Clinical evaluation scores by KG.
Clinical evaluation scores by QL.
Comparison and bidirectional encoder representations from transformers (BERT) scores of World Health Organization answers and ChatGPT 3.5 and 4.0 versions.
Generated responses by ChatGPT 3.5 and ChatGPT 4.0 for COVID-19 question and answer.
AbbreviationsAI
artificial intelligence
BERT
Bidirectional Encoder Representations from Transformers
Q&A
question and answer
WHO
World Health Organization
The authors are deeply indebted to the advancements in machine learning and artificial intelligence for bolstering the methodological framework of this study. Specifically, the authors used the ChatGPT 3.5 and 4.0 language models to autonomously generate the questions that served as the cornerstone of our evaluation metrics. The generated text and prompt words from these models can be found in Multimedia Appendices 2 and 5, respectively. Concurrently, we used bidirectional encoder representations from transformers (BERT) algorithms for the quantitative evaluation of text quality. Detailed metrics, including BERT scores, are available in Multimedia Appendix 5. This computational approach underwent rigorous statistical scrutiny, which was instrumental in enhancing both the analytical rigor and methodological precision of our research. The “nonhuman assistance” provided by these advanced algorithms was indispensable in elevating the academic quality of our study. The study received funding from several sources. The National Natural Science Foundation of China (grants 30973440 and 30770950) supported the data collection, analysis, and interpretation. The Ministry of Key Laboratory of Child Development and Disorders provided funding through the Youth Basic Research Project (grant YBRP-2021XX). Additionally, the preparation of the paper was funded by key projects of the Chongqing Natural Science Foundation, specifically grants cstc2020jcyj-msxmX0326 and CSTB2022NSCQ-MSX0819. The funding agency paid for the scholarships of the students involved in the research.
Data Availability
All data generated or analyzed during this study are included in this published paper and its supplementary information files (Multimedia Appendices 1-6).
GW played a pivotal role in the conceptualization of the study. Both GW and YW were responsible for data curation, ensuring the accuracy and organization of the data collected. The formal analysis of the data was a collaborative effort by GW, WZ, and KZ. KG and QL conducted the investigation and contributed to the collection and interpretation of research data. The validation of the study's findings and methodologies was carried out by KG, QL, WZ, and KZ, ensuring the reliability and accuracy of the results. The original draft of the paper was written by GW and KZ, where they articulated the study's findings and significance. Finally, all authors participated in reviewing and editing the paper, contributing their insights and expertise, and they all approved the paper for submission.
None declared.
JiangFDengLZhangLCaiYCheungCWXiaZReview of the clinical characteristics of coronavirus disease 2019 (COVID-19)J Gen Intern Med20203551545154910.1007/s11606-020-05762-w3213357810.1007/s11606-020-05762-wPMC7088708WHO Coronavirus (COVID-19) dashboardWorld Health Organization20202020-06-01https://covid19.who.int/RamanRPatelKJRanjanKCOVID-19: unmasking emerging SARS-CoV-2 variants, vaccines and therapeutic strategiesBiomolecules202111799310.3390/biom1107099334356617biom11070993PMC8301790BrunsDPKraguljacNVBrunsTRCOVID-19: facts, cultural considerations, and risk of stigmatizationJ Transcult Nurs202031432633210.1177/104365962091772432316872PMC7324134DoraiswamySCheemaSMaisonneuvePAbrahamAWeberIAnJLowenfelsABMamtaniRKnowledge and anxiety about COVID-19 in the State of Qatar, and the Middle East and North Africa region-a cross sectional studyInt J Environ Res Public Health20211812643910.3390/ijerph1812643934198649ijerph18126439PMC8296266AdilMTRahmanRWhitelawDJainVAl-TaanORashidFMunasingheAJambulingamPSARS-CoV-2 and the pandemic of COVID-19Postgrad Med J202197114411011610.1136/postgradmedj-2020-13838632788312postgradmedj-2020-138386PMC10016996PradhanMShahKAlexanderAAjazuddinMinzSSinghMRSinghDYadavKChauhanNSCOVID-19: clinical presentation and detection methodsJ Immunoassay Immunochem2022431195129110.1080/15321819.2021.195129134355645FangELiuXLiMZhangZSongLZhuBWuXLiuJZhaoDLiYAdvances in COVID-19 mRNA vaccine developmentSignal Transduct Target Ther2022719410.1038/s41392-022-00950-y3532201810.1038/s41392-022-00950-yPMC8940982HosseiniESKashaniNRNikzadHAzadbakhtJBafraniHHKashaniHHThe novel coronavirus disease-2019 (COVID-19): mechanism of action, detection and recent therapeutic strategiesVirology20205511910.1016/j.virol.2020.08.01133010669S0042-6822(20)30172-0PMC7513802RollettRCollinsMTamimyMSPerksAGBHenleyMAshfordRUCOVID-19 and the tsunami of informationJ Plast Reconstr Aesthet Surg202174119920210.1016/j.bjps.2020.08.11233645504S1748-6815(20)30461-7PMC7502168RovettaABhagavathulaASCOVID-19-related web search behaviors and infodemic attitudes in Italy: infodemiological studyJMIR Public Health Surveill202062e1937410.2196/1937432338613v6i2e19374PMC7202310WatkinsRGuidance for researchers and peer-reviewers on the ethical use of Large Language Models (LLMs) in scientific research workflowsAI Ethics20231610.1007/s43681-023-00294-5ChartrandGChengPMVorontsovEDrozdzalMTurcotteSPalCJKadourySTangADeep learning: a primer for radiologistsRadiographics20173772113213110.1148/rg.201717007729131760BoßelmannCMLeuCLalDAre AI language models such as ChatGPT ready to improve the care of individuals with epilepsy?Epilepsia20236451195119910.1111/epi.1757036869421YazdaniACostaSKroonBArtificial intelligence: friend or foe?Aust NZ J Obst Gynaecol202363212713010.1111/ajo.13661AbdulaiAFHungLWill ChatGPT undermine ethical values in nursing education, research, and practice?Nurs Inq2023303e1255610.1111/nin.1255637101311TekinayONCurious questions about Covid-19 pandemic with ChatGPT: answers and recommendationsAnn Biomed Eng20235171371137310.1007/s10439-023-03209-x3707128210.1007/s10439-023-03209-xPMC10112300TemsahMHJamalAAl-TawfiqJAReflection with ChatGPT about the excess death after the COVID-19 pandemicNew Microbes New Infect20235210110310.1016/j.nmni.2023.10110336844147S2052-2975(23)00022-7PMC9937725LeeJYoonWKimSKimDKimSSoCKangJBioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics20203641234124010.1093/bioinformatics/btz682315018855566506PMC7703786SallamMChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concernsHealthcare (Basel)202311688710.3390/healthcare1106088736981544healthcare11060887PMC10048148PatelSBLamKChatGPT: the future of discharge summaries?Lancet Digit Health202353e107e10810.1016/S2589-7500(23)00021-336754724S2589-7500(23)00021-3Milne-IvesMde CockCLimEShehadehMHde PenningtonNMoleGNormandoEMeinertEThe effectiveness of artificial intelligence conversational agents in health care: systematic reviewJ Med Internet Res20202210e2034610.2196/2034633090118v22i10e20346PMC7644372CascellaMMontomoliJBelliniVBignamiEEvaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenariosJ Med Syst20234713310.1007/s10916-023-01925-43686992710.1007/s10916-023-01925-4PMC9985086LeeTCStallerKBotomanVPathipatiMPVarmaSKuoBChatGPT answers common patient questions about colonoscopyGastroenterology20231652509511.e710.1053/j.gastro.2023.04.03337150470S0016-5085(23)00704-7ShenYHeacockLEliasJHentelKDReigBShihGMoyLChatGPT and other large language models are double-edged swordsRadiology20233072e23016310.1148/radiol.23016336700838SallamMSalimNAAl-TammemiABBarakatMFayyadDHallitSHarapanHHallitRMahafzahAChatGPT output regarding compulsory vaccination and COVID-19 vaccine conspiracy: a descriptive study at the outset of a paradigm shift in online search for informationCureus2023152e3502910.7759/cureus.3502936819954PMC9931398CoxASethIXieYHunter-SmithDJRozenWMUtilizing ChatGPT-4 for providing medical information on blepharoplasties to patientsAesthet Surg J2023438NP658NP66210.1093/asj/sjad096370325217111402BeltramiEJGrant-KelsJMConsulting ChatGPT: ethical dilemmas in language model artificial intelligenceJ Am Acad Dermatol2023S0190S962210.1016/j.jaad.2023.02.05236907556S0190-9622(23)00364-XSebastianGDo ChatGPT and other AI Chatbots pose a cybersecurity risk?: An exploratory studyInt J Secur Priv Pervasive Comput202315111110.4018/ijsppc.320225SebastianGPrivacy and data protection in ChatGPT and other AI Chatbots: strategies for securing user informationSSRN J2023445476110.2139/ssrn.4454761DeneckeKAbd-AlrazaqAHousehMArtificial intelligence for chatbots in mental health: opportunities and challengesMultiple Perspectives on Artificial Intelligence in Healthcare: Opportunities and Challenges2021Cham, GermanySpringer International PublishingRosenbaumPRRubinDBThe central role of the propensity score in observational studies for causal effectsBiometrika1983701415510.1093/biomet/70.1.41TanKLLeeCPAnbananthenKSMLimKMRoBERTa-LSTM: a hybrid model for sentiment analysis with transformer and recurrent neural networkIEEE Access202210215172152510.1109/access.2022.3152828BiswasSSRole of Chat GPT in public healthAnn Biomed Eng202351586886910.1007/s10439-023-03172-73692057810.1007/s10439-023-03172-7