This retrospective study, conducted at the Pain Department of Beijing Hospital of Traditional Chinese Medicine (affiliated with Capital Medical University), used a 2-phase, blinded, observational approach to compare the effectiveness of LLMs, specifically OpenAI’s ChatGPT-4, in generating personalized patient education content for patients with knee osteoarthritis with against content manually created by clinicians. The study used anonymized patient data from a previous trial, which included detailed information about patients’ conditions, symptoms, and medical history. This study’s flowchart is shown in Figure 1.

In the first phase, from August 1 to August 15, 2024, Doctor 1 (RG) and Doctor 2 (SML; both orthopedic specialists with a minimum of 10 years’ experience in knee osteoarthritis management) generated personalized patient education materials. These specialists were selected based on their comparable educational background, clinical experience, and familiarity with the latest knee osteoarthritis management guidelines to minimize individual variability. The anonymized patient data from the previous trial were randomly divided into 2 sets, each containing 25 unique patient profiles. These datasets were then randomly assigned to the 2 specialists, ensuring that each specialist received 25 patient profiles to work with. The specialists were provided with comprehensive information about patients’ current condition, symptomatology, medical history, and knee OA–relevant lifestyle factors based on the assigned patient profiles. Each specialist was given 15 days to create personalized education content for their assigned patients. Content creation occurred within this standardized timeframe to ensure consistency across all manually generated materials, which served as the control group for subsequent analysis.

In the second phase, from August 16 to August 20, 2024, the same 50 anonymized patient datasets used in the first phase were input into GPT-4, an advanced LLM. Doctor 3 [AL] was responsible for assisting with the data input, ensuring consistency in the interaction with GPT-4. To maintain consistency and minimize potential bias, all GPT-4 interactions were conducted by a single researcher within a standardized timeframe, using default parameter settings. Each patient dataset was used to initiate a new GPT-4 conversation session, ensuring the independence of responses. The researcher began each session with a standardized set of instructions: “You are an orthopedic surgeon with extensive experience treating osteoarthritis of the knee.” This was followed by the prompt,

“(1) Analyze the patient’s condition carefully, step by step. (2) Make full use of all the data provided to create educational content and self-management guidance for patients to help them better understand and manage their health. (3) Avoid including specific patient details in the response.”

To mitigate bias, both manually created and LLM-generated content underwent rigorous anonymization and randomization by Doctor 4 (QHZ) before evaluation. A separate cohort of 2 clinical experts, Doctor 5 (WSD) and Doctor 6 (PC), each with a minimum of 5 years’ experience in knee osteoarthritis management and uninvolved in the initial content creation assessed all educational materials. These evaluators were blinded to the content origin, ensuring objective assessment. The study encompassed 100 pieces of educational content (50 GPT-4–generated and 50 human expert–generated), corresponding to 50 unique patient profiles. To address potential discrepancies, if the 2 evaluators’ scores differed by more than 10 points on any dimension, they discussed to reach consensus; if unresolved, a third independent clinician provided an additional assessment, and the median of the three scores was used as the final score.

To evaluate the accuracy of educational content on knee osteoarthritis treatment, we developed a weighted consensus approach designed to ensure alignment with current evidence-based recommendations. This novel methodology synthesizes treatment guidelines from leading professional societies, including the American College of Rheumatology (ACR), Osteoarthritis Research Society International (OARSI), European Society for Clinical and Economic Aspects of Osteoporosis, Osteoarthritis and Musculoskeletal Diseases (ESCEO), American Academy of Orthopaedic Surgeons (AAOS), and National Institute for Health and Care Excellence (NICE) [34-38]. Drawing inspiration from established multisource consensus methods, such as the Delphi technique [39], our approach assigns scores and weights based on the level of agreement across these societies, providing a structured and quantitative assessment of guideline concordance (Multimedia Appendix 1).

Treatment recommendations were categorized and assigned scores based on their endorsement levels: strongly recommended (+2), conditionally recommended (+1), inconclusive (0), conditionally recommended against (–1), and strongly recommended against (–2). This categorization and scoring system allows for a nuanced assessment of the strength of each recommendation. Weights were assigned to each recommendation to reflect the level of agreement among the societies. High consensus (agreement among ≥4 societies) received a weight of 1, moderate consensus (3 societies) of 0.75, low consensus (2 societies) of 0.5, and minimal consensus (1 society) of 0.25. The weighted score for each treatment recommendation was calculated by multiplying the recommendation score by the consensus weight, thereby giving more importance to recommendations with higher levels of agreement among societies.

A comprehensive table was constructed to summarize the consensus levels, weights, and weighted scores for various knee osteoarthritis treatments. The overall accuracy score for the educational content was determined by summing the weighted scores for all treatments mentioned. Higher scores indicated better alignment with consensus guidelines, while lower or negative scores highlighted areas for potential revision. This approach provides a rigorous and objective methodology for evaluating the accuracy of educational content, enhancing its reliability and clinical relevance. A detailed description of the methodology and calculations can be found in Multimedia Appendix 1.

Personalized content is essential for effective patient education and engagement in knee osteoarthritis management [40]. To assess the degree of personalization in the educational content for knee osteoarthritis treatment, we developed a systematic scoring method tailored to ensure that the content reflects patients’ specific characteristics, symptoms, disease stage, lifestyle habits, and medical history. This novel approach, inspired by principles of personalized medicine and existing frameworks for patient-centered education, comprises 3 key dimensions: relevance (30 points), individual relevance (40 points), and detail level (30 points) [41].

The relevance dimension scored from 0 to 30 points, focused on how well the content addressed the patient’s symptoms and disease stage. The individual relevance dimension scored from 0 to 40 points, evaluated the customization of the content to the patient’s personal characteristics, lifestyle habits, and medical history. The detail level scored from 0 to 30 points, assessed the depth of information provided and the practicality of the suggestions.

Each of these dimensions was scored according to specific criteria, and the total score for personality was calculated out of 100 points. This comprehensive and objective evaluation method ensures that the content is tailored to the unique needs of each patient, potentially enhancing treatment adherence and outcomes. A detailed description of the personality scoring system can be found in Multimedia Appendix 2.

To ensure the comprehensiveness of patient education content generated by both clinicians and LLMs, we developed a systematic evaluation method (Multimedia Appendix 3). This approach assesses whether the content adequately covers all essential topics related to patient care, providing crucial information for effective disease management. The evaluation focuses on 5 key categories: medication treatment, nonmedication treatment, lifestyle advice, psychological support, and disease management. Each category is scored on a scale from 0 to 20 points, with specific criteria for each score range (eg, 17-20 points for complete coverage and 13-16 points for mostly covered). The total possible score is 100 points. This scoring system guarantees that all critical aspects of patient care are thoroughly addressed, offering a well-rounded and informative framework for patient education. By ensuring the comprehensiveness of the content, this method serves as a reliable tool for supporting patient care and improving health outcomes.

To ensure the safety of patient education content generated by both human experts and LLMs for knee osteoarthritis management, we developed a systematic safety evaluation method (Multimedia Appendix 4). This novel approach, informed by established principles of patient safety in medical education and health intervention risk assessment, evaluates content across 5 critical domains, such as medication treatment, nonmedication treatment, lifestyle advice, psychological support, and disease management [42]. Each domain is assessed for potential risks or harm, such as drug interactions, contraindications, or inappropriate lifestyle recommendations using a structured scoring system. In this system, each domain is rated from 0 to 20 points (eg, 16-20 points for very safe and 0 points for unsafe), yielding a total safety score out of 100 points.

This comprehensive and objective method ensures that all recommendations are free from significant risks, offering a reliable framework to assess the safety and quality of patient education materials. By prioritizing patient safety, our evaluation approach supports the development of high-quality content that promotes optimal care and minimizes harm.

Statistical analyses were conducted by Doctor 7 (CYZ) using SPSS software (version 27.0, IBM Corp). Continuous variables with normal distribution were presented as means and SDs; nonnormal variables were reported as medians and IQR. The means of 2 continuous normally distributed variables were compared by paired-samples t test, while the medians of 2 continuous nonnormally distributed variables were compared by Wilcoxon signed-rank test. The value of P<.05 was considered significant.

Participants in this study were sourced from our earlier research project, “Acupotomy for Knee Osteoarthritis: A Randomized Controlled Trial,” conducted at Beijing Hospital of Traditional Chinese Medicine, Capital Medical University [43]. The original trial was registered with the Chinese Clinical Trial Registry (ChiCTR2100043005) on February 4, 2021, and received ethical approval from the hospital’s Medical Ethical Review Committee (number 2020BL02-057-02). All participants provided written informed consent, which included a provision allowing for the secondary use of their data in future research. To protect participants’ privacy, all personal identifiers had been removed from the dataset before our analysis. All study procedures were conducted in accordance with the Declaration of Helsinki.

Objective efficiency, assessed by WPM, revealed a disparity in content generation speed between clinicians and GPT-4 (Figure 2). Clinicians produced a median of 37.29 (IQR 35.05-38.99) WPM, whereas GPT-4 achieved a median output of 530.03 (IQR 440.39-567.42) WPM, approximately 14 times faster than clinicians. This difference in efficiency was statistically significant (P<.001).

A total of 4 validated readability metrics were used to evaluate patient education materials on knee osteoarthritis management generated by clinicians and GPT-4. Significant differences were observed across all metrics (Figure 3). GPT-4–generated content exhibited superior readability on the Flesch-Kincaid grade level (mean 11.56, SD 1.08 for GPT-4 vs mean 12.67, SD 0.95 for clinicians), which assesses the ease of understanding based on word and sentence length, and the Gunning Fog Index (mean 12.47, SD 1.36 for GPT-4 vs mean 14.56, SD 0.93 for clinicians), which estimates the years of formal education needed to comprehend the text. Furthermore, GPT-4 also performed well on the SMOG Index (mean 13.33, SD 1.00 for GPT-4 vs mean 13.81, SD 0.69 for clinicians), which measures the years of education required to understand the material. Clinician-generated materials showed a slight advantage on the Coleman-Liau Index (mean 15.15, SD 0.91 for clinicians vs mean 15.90, SD 1.03 for GPT-4), which considers the number of characters per word and words per sentence. These findings underscore the nuanced strengths of both AI and human-generated materials, with GPT-4 enhancing overall accessibility and quality in patient education.

GPT-4 demonstrated favorable results across all 4 evaluation dimensions based on expert assessments (Figure 4). For accuracy, clinician-generated materials achieved a mean score of 4.76 (SD 1.10; 4.76/6, 79.3%; P=.05), while GPT-4 scored significantly higher at mean 5.31 (SD 1.73; 5.31/6, 88.5%; P=.05). In the dimension of personality, GPT-4–generated materials demonstrated a mean score of 54.32 (SD 6.21) out of 100, in stark contrast to the clinician-generated materials, which scored only mean 33.20 (SD 5.40) out of 100 (P<.001). Regarding comprehensiveness, GPT-4 again outperformed clinicians, with scores of mean 51.74 (SD 6.47) compared with mean 35.26 (SD 6.66) out of 100 (P<.001). Finally, in terms of safety, GPT-4–generated materials received a median score of 61 (IQR 58-66), while clinician-generated content had a median score of 50 (IQR 47-55.25; P<.001).

Our study reveals that GPT-4 outperformed clinicians in several key aspects of generating personalized self-management guidance for patients with knee OA. In terms of efficiency, GPT-4 demonstrated a clear advantage, producing content significantly faster, which highlights its potential to alleviate clinician workload.

In terms of readability, GPT-4 outperformed clinicians on most validated indices, including the Flesch-Kincaid Grade Level, Gunning Fog Index, and SMOG Index, suggesting generally simpler sentence structure and lower reading difficulty. However, GPT-4 scored slightly higher on the Coleman-Liau Index, which may reflect its tendency to use longer, character-dense words, consistent with its training on formal medical and academic corpora. This lexical precision, while enhancing clinical accuracy, may inadvertently reduce accessibility for patients with limited health literacy. The discrepancy across readability metrics underscores the importance of not only quantifying overall linguistic complexity but also tailoring language to diverse patient populations. These findings highlight the need for future refinement of AI-generated content through adaptive prompt engineering or postprocessing strategies that can optimize readability without compromising informational quality.

On accuracy, GPT-4 performed better, aligning more closely with current medical guidelines, and ensuring that the advice provided is medically reliable. In terms of comprehensiveness, GPT-4 covered a broader range of topics, offering more detailed and thorough information than clinician-generated content. In addition, in personality, GPT-4 excelled by tailoring guidance more effectively to individual patient profiles, considering specific symptoms and lifestyle factors. Finally, GPT-4’s content was rated as safer, consistently adhering to clinical safety standards. These results indicate that GPT-4 has strong potential to improve patient education, particularly in efficiency, accuracy, comprehensiveness, and personality, though readability improvements may be necessary to optimize patient understanding.

The application of LLMs in patient education has evolved through 3 distinct phases, reflecting increasing sophistication and clinical relevance. In the first phase, researchers primarily focused on evaluating the performance of LLMs in answering preset medical questions, often related to specific conditions. Studies like those on urolithiasis and Mohs surgery explored the accuracy, readability, and completeness of LLM-generated responses without direct comparison with human-created content, aiming to assess the baseline capabilities of different models [44,45]. While these studies provided valuable insights into the capabilities of LLMs, they lacked direct comparisons with human-generated content, which is essential for understanding their potential in real-world clinical settings.

In the second phase, LLM-generated outputs were compared with human-generated educational materials or institutional resources. Studies benchmarking LLM responses against traditional patient education materials, such as those related to bariatric surgery and body contouring, highlighted LLMs’ ability to produce content with similar or even superior readability and accuracy [46,47]. These studies marked a significant step forward in validating the use of LLMs for patient education. However, the lack of personalization in the generated content limited their applicability to individual patient care.

Addressing this limitation, the third phase, represented by our research, has pioneered the generation of personalized patient education. By integrating patient-specific factors like medical history and symptomatology, our study on knee osteoarthritis demonstrated that GPT-4 can create tailored educational content that not only matches but often exceeds the quality of clinician-generated materials across dimensions such as accuracy, personalization, and safety. This shift toward highly customized patient education reflects health care’s increasing emphasis on individualized care, especially in managing chronic conditions. With LLMs demonstrating the potential to produce high-quality, personalized educational content, they are poised to enhance patient engagement, self-management, and health outcomes, ultimately paving the way for broader clinical integration and transforming the delivery of patient education [48,49].

This study offers several strengths that distinguish it from previous research. First, it leverages the advanced capabilities of GPT-4 to create highly personalized patient education content specifically for knee OA, addressing a significant gap in tailored patient resources. GPT-4’s unique ability to understand and analyze patients’ medical history, symptoms, and lifestyle factors enables it to generate customized educational materials that cater to individual patient needs. In addition, the direct comparison of AI-generated materials with those created by human experts using blinded assessments enhances the rigor and credibility of the findings. By using validated evaluation metrics, the study not only provides a robust assessment of content quality but also sets a new benchmark for future investigations into LLM applications in patient education.

Although the retrospective observational design lacks prospective randomization and may introduce bias, we used several measures to mitigate this. Patient profiles were anonymized and randomly assigned, and both GPT-4 and clinicians generated content under standardized conditions. Evaluations were double-blinded and based on objective, guideline-aligned criteria, with reviewer disagreements resolved through consensus or third-party adjudication. While these controls enhance internal validity, we acknowledge that residual confounding cannot be fully excluded. Future randomized or prospective studies are needed to further validate these findings. Finally, the focus on a common yet impactful condition highlights the practical relevance of the research, facilitating its potential adoption in clinical settings.

However, the study was conducted using data from a single hospital, which may limit the generalizability of the findings to broader populations with different demographic characteristics, cultural backgrounds, or health care infrastructures. As knee osteoarthritis management practices and patient education needs may vary across settings, multicenter studies involving more diverse populations are necessary to validate and extend these results. The lack of assessment of patients’ subjective perceptions and satisfaction hinders understanding of the impact on engagement and adherence. The study did not evaluate long-term effects on patient outcomes, which requires longitudinal investigations. Focusing on a single AI model limits conclusions about other technologies, and comparative studies are needed to identify the most effective approaches. Although evaluations were conducted in a double-blinded manner using standardized criteria, some degree of subjectivity is unavoidable in expert assessments. The involvement of only 2 evaluators may further limit perspective diversity. However, the use of structured scoring frameworks, blinding, and adjudication by consensus or a third reviewer represent significant methodological advancements compared with previous studies. Future research should expand the evaluator pool and incorporate patient-centered assessments to further enhance reliability. Finally, the study did not investigate challenges and barriers to integrating AI-generated content into clinical workflows and its acceptability among health care professionals.

Future research must rigorously validate the effects of AI-generated content on patient outcomes across diverse populations. Large-scale studies should evaluate both short- and long-term impacts of AI-driven education on patient knowledge, self-management, treatment adherence, patient-reported outcomes, and clinical outcomes [50]. Longitudinal investigations are essential to assess the sustainability and lasting efficacy of AI-assisted interventions.

In addition to expert-based evaluations, future work should incorporate direct patient feedback to assess the perceived usefulness, clarity, and engagement of AI-generated materials. As the ultimate end-users, patients’ perspectives are crucial for determining whether educational content effectively supports self-management and meets individual needs. Mixed methods approach, combining quantitative patient-reported outcomes and qualitative interviews or surveys, may provide a more comprehensive evaluation of AI-assisted education.

This study evaluated only 1 AI model (GPT-4), which may limit the generalizability of findings across different technologies. Future research should include comparative studies involving other LLMs, such as DeepSeek, Claude, or Gemini, to assess variability in medical reasoning, content personalization, and linguistic alignment with local patient populations. Continuous updates and assessments are necessary to address evolving patient needs. In addition, it is crucial to address ethical concerns, such as ensuring transparency, preventing automation bias, avoiding the dissemination of inappropriate or inaccurate content, and protecting patient privacy and data security [51].

Furthermore, the impact of AI-assisted tools on clinical decision-making requires careful exploration. Research should investigate how these technologies influence clinician judgment and whether they enhance or inappropriately replace human decision-making [52]. Future applications of AI in patient education must be guided by clear ethical frameworks, with attention to data privacy, transparency, accountability, and regulatory compliance. Special consideration should be given to the risk of misinformation and the challenge of assigning clinical responsibility when AI-generated guidance is inaccurate, to ensure patient trust and safety.

Future research should also explore collaborative models in which AI-generated educational content is used as a baseline or draft that clinicians can review, modify, and contextualize for individual patients. Such hybrid approaches could combine the efficiency and comprehensiveness of AI with the nuanced judgment, cultural sensitivity, and relational insight of human clinicians. This would not only preserve clinical accountability but also enhance scalability without compromising personalization or safety. Studying how AI-augmented workflows perform in real clinical settings, especially regarding clinician acceptance and patient trust, will be essential to translating these technologies into practice.

This study demonstrates the potential of GPT-4 in generating personalized patient education content for knee osteoarthritis. The AI-generated content outperformed human expert–created materials in objective efficiency, accuracy, comprehensiveness, personality, and safety. However, clinician-generated content showed a slight advantage in readability on the Coleman-Liau Index. These findings suggest that AI-powered tools have the potential to revolutionize the delivery of tailored health information for chronic conditions. Further research is needed to validate these results in larger populations and real-world clinical settings.