Patient education materials (PEMs) are key tools for improving patients’ health literacy, enhancing treatment adherence, and improving clinical outcomes [1-3]. However, the effectiveness of PEMs is largely limited by their readability [4-9]. The National Institutes of Health (NIH) recommends that medical information be written at a sixth-grade reading level to accommodate the health literacy of the general population [10]. Unfortunately, due to the overuse of technical terminology, traditional manually written materials often fail to meet this standard and are costly, time-consuming, and labor-intensive, which limits access to information and exacerbates health inequalities [11-13].

In recent years, breakthroughs in generative AI (GenAI) have opened up new opportunities to address this challenge [11,14-17]. Technologies such as large language models (LLMs), with their powerful natural language processing capabilities, can efficiently generate structured, personalized medical content [18,19]. As tools like ChatGPT and Google Gemini become increasingly widely adopted by the public, more and more patients and clinicians are beginning to use LLMs to obtain health information or assist in creating educational materials [15,16,20]. Evidence suggests that LLMs have great potential in simplifying text and explaining complex treatment regimens, and are expected to alleviate the limitations of traditional materials such as high costs and long update cycles [21-23].

Despite rapid development, the efficacy and safety of LLMs in generating PEMs remain controversial [24-27]. Most existing studies focus on a single disease or a specific model, with inconsistent results [28-30]. Some studies indicate that model-generated content performs well in terms of technical accuracy, but often has problems such as poor readability, insufficient actionable guidance, and the potential risk of hallucinations [28,31-33]. Patients generally have low digital health literacy and are more accustomed to directly accessing the internet or using LLMs to search for health knowledge, rather than actively consulting evidence-based clinical guidelines [28,31-33]. Furthermore, there is currently a lack of systematic reviews of different disease areas, model performance, and the use of evaluation tools, making it difficult to reach a unified consensus on clinical application [34-36]. Notably, current evaluations usually rely mainly on expert judgment or generic metrics, while patient involvement is still largely lacking [32,37].

Although prior reviews have explored the broader application of LLMs in patient education, they have not systematically focused on the generation of new PEMs. The emerging evidence base for their performance in different clinical scenarios remains fragmented. Given the heterogeneity of the literature in terms of study designs, evaluation methods, and reporting depth, we conducted a scoping review to map the landscape rather than to estimate pooled effect sizes.

The primary objective of this review was to synthesize the published literature on assessing the performance of new PEMs generated by LLMs. The specific objectives were to (1) characterize the clinical scenarios or core educational tasks addressed by these materials; (2) identify the information assessment tools used to evaluate their readability, quality, and usability; and (3) synthesize the findings reported across these evaluative dimensions and compare the performance of LLMs. These findings are expected to provide a basis for improving the design, evaluation, and implementation of LLMs, and for promoting more consistent and comparable assessment across future studies.

This scoping review was designed to systematically map and synthesize evidence on the readability, quality, and usability of LLM-generated new PEMs across clinical settings. The review followed the methodological framework proposed by Arksey and O’Malley [38] and was reported in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) [39] guidelines, and a completed checklist is provided in Multimedia Appendix 1. The review protocol was registered in the Open Science Framework (OSF registration number: EN425).

We conducted a systematic search in PubMed, Embase, the Cochrane Library, Web of Science, and Scopus. The searches were limited to peer-reviewed papers published between January 1, 2000, and January 1, 2026. We also conducted backward and forward citation tracking for the included studies. The final search was completed on January 25, 2026. Additionally, grey literature databases were not searched because they may lack complete information on study design or data collection, and are not necessarily peer-reviewed, which could compromise the standardization of data extraction and the reliability of comparisons across studies.

The search strategy was developed and refined by 2 researchers (CL and YC) with the assistance of a research librarian. It was structured around 2 core concepts: LLMs and PEMs. Controlled vocabulary and free-text terms were combined using Boolean operators. The search syntax and concept-block structure were adapted to the indexing system and functionality of each database. Apart from publication-date limits, no filters were applied for language, age, or publication type at the search stage. The full search strategy is provided in Multimedia Appendix 2.

Eligibility criteria were defined a priori using the population, concept, and context (PCC) framework [40] recommended for scoping reviews (Textbox 1). In this review, the population referred to implemented LLMs relevant to health care, the concept referred to new PEMs generated by LLMs and its evaluation, and the context referred to any health care setting. Original studies published in English that evaluated new PEMs generated by LLMs in health care using information assessment tools were included, provided that the full text was available. The exclusion of literature other than English is due to the significant differences in syllabic structure, orthography, and syntax between different languages, which may affect the consistency of evaluation criteria and the comparability of evaluation results.

One researcher (CL) conducted the literature search. All retrieved records were imported into NoteExpress, and duplicate entries were removed. Two researchers (CL and XL) independently screened the titles and abstracts to assess whether they met the inclusion criteria. Records judged potentially eligible or unclear by either researcher proceeded to full-text assessment. The same 2 researchers (CL and XL) then independently conducted a detailed assessment based on the eligibility criteria. Reasons for exclusion at the full-text stage were documented and reported in the PRISMA flowchart. Disagreements at any stage of the screening process were resolved by consultation with a senior researcher (ML).

Two independent reviewers (CL and YC) developed a standardized data extraction form in Microsoft Excel. The form was piloted on 3 randomly selected included studies and refined to ensure consistent interpretation of variables and coding rules. One researcher (CL) initially was responsible for data charting, and another researcher (JL) independently verified the extracted data for accuracy and consistency. Any discrepancies were resolved through discussion, and a third researcher was consulted when necessary (ML).

The data extracted from each included study included title, publication year, journal, first author, name and version of the LLMs, input instructions of LLMs, diseases, evaluation indicators, evaluator characteristics, data collection time, and main conclusions. We also extracted the information assessment tools used in each study such as DISCERN [41]. Although the assessment tools varied across studies, most evaluated at least one of the following domains: (1) readability, including language difficulty and reading fluency; (2) quality, including the reliability, accuracy, and completeness of the generated content; and (3) usability, including the understandability and actionability of PEMs. All tools used to evaluate these domains and their corresponding results in each study were extracted completely.

In accordance with scoping review methodology, we did not conduct a formal methodological quality appraisal or risk-of-bias assessment. This study was designed to map the range and characteristics of the available evidence rather than to evaluate intervention effects or the strength of evidence outcomes. Although no formal appraisal was performed, this limitation was acknowledged and considered when interpreting the findings.

All charted data were synthesized using descriptive and narrative methods. The calculation of frequencies and percentages was performed using Microsoft Excel. To visually present evaluation gaps, we classified the PEMs evaluated in each included study and compared their readability, quality, and usability. Findings were summarized using tables and figures to support interpretability and transparency.

The study screening process is summarized in the PRISMA-ScR flow diagram (Figure 1). A total of 1113 records were retrieved from the 5 databases. After removing 619 duplicates, 494 records were screened for titles and abstracts, of which 405 were excluded. The remaining 89 reports were sought for retrieval, and 4 were not retrieved. Following full-text review, 30 studies [42-71] met the inclusion criteria.

As shown in Table 1, a total of 5 studies [52-54,67,68] evaluated a single model, while 25 studies [42-51,55-66,69-71] compared multiple models. Included studies covered 11 different LLM families. All studies included models from the ChatGPT family; the Google Gemini family appeared in 14 studies [42,45-47,49-51,59-62,64,65,69], and the DeepSeek family appeared in 6 studies [43,45,56-58,71]. Other evaluated models included Google Bard (n=3), Anthropic Claude (n=2), Meta Llama (n=2), and Microsoft Copilot (n=2).

Regarding the instruction, 22 studies (73%) [42-45,48,49,51-57,59-64,68,70,71] used a single prompt, such as “Generate patient education materials for a specific disease,” while 8 studies (27%) [46,47,50,58,65-67,69] used a predefined list of questions, ranging from 5 to 35 questions. Overall, 23 studies (77%) [44,46-54,56-58,60-62,64-70] reported the number of researchers, ranging from 1 to 12, and no patient involvement. A total of 22 studies (73%) [42-44,46-48,50,51,53,55-62,64,66,69-71] provided data collection dates, ranging from October 2023 to May 2025.

The range of diseases covered included more than 30 conditions such as ophthalmology (n=8), orthopedics (n=5), neurological disorders (n=4), and metabolic disorders (n=4). The main findings indicated that most studies reported that LLM-generated content generally exceeded the recommended sixth-grade reading level. All studies emphasized that LLMs must be used in conjunction with professional review and cannot fully replace medical resources.

Similarly, we extracted all data on usability assessment metrics in Multimedia Appendix 5; a total of 18 records included usability assessments [44,46,47,49,50,53,54,56,60-62,64,67], all of which used the Patient Education Materials Assessment Tool (PEMAT) or Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P) scales. Based on the classification of PEMs, we present the usability levels of 5 major categories of PEMs, as shown in Figure 7.

Overall, most PEMs performed well in terms of understandability, but there were widespread shortcomings in terms of actionability. Specifically, perioperative patient education performed best, with 2 records rated as excellent for both understandability and actionability [54,67]. Specialty disease diagnostic and therapeutic education was primarily rated as excellent for understandability, but rated as poor in actionability in many cases. Chronic disease management education generally scored higher on understandability than on actionability, with most records rated as excellent in understandability. Pharmacotherapy safety and adherence education showed balanced performance, with all records rated as good in actionability. For public health and preventive education, both understandability and actionability were mostly rated as good, though 2 records were rated as poor in actionability [53,61].

In this study, we conducted a comprehensive assessment of the current state of LLMs in generating new PEMs. We found that the LLMs in this field are rapidly expanding, driven primarily by the ChatGPT family of models. Although LLMs have demonstrated the ability to generate well-structured content, there is a significant disparity between the quality of their outputs and established health literacy standards. Specifically, only 10% of the included studies reached the reading level recommended by the NIH, and materials in most disease categories consistently lacked actionability. Furthermore, only half of the studies used standardized usability assessment tools, highlighting the fragmentation of current PEMs evaluation systems.

Our study suggests that while LLMs improve the efficiency of developing PEMs, without optimization, direct application of LLMs may inadvertently exacerbate health inequities, because the generated materials are often too complex for patients to understand or translate into healthy behaviors.

Aligning the readability of PEMs with patients’ health literacy is the foundation to achieve health benefits [73]. Although the NIH recommends a sixth-grade reading level, we found that the vast majority of LLM-generated materials exceeded this threshold [28,29,31,36]. Notably, we observed significant heterogeneity across different disease categories. Specialty disease diagnostic and therapeutic education demonstrated the best readability, with most studies falling within the 7th-grade to high school level, while chronic disease management education had poor readability. This discrepancy may stem from the inherent complexity of chronic disease management, which involves subtle lifestyle changes and multifactorial decision-making, making it difficult for LLMs to develop PEMs without losing critical context [74-76]. Ma et al [77] presented a similar conclusion.

We believe that the readability dilemma stems less from limitations in the model capabilities and more from a lack of scientifically sound implementation strategies [78,79]. Excessively high reading difficulty not only hinders access to information but may also exacerbate health inequalities among groups with lower socioeconomic status [16,17]. Future developments should not focus solely on improving model performance or relying on default outputs; instead, they must use targeted prompt engineering to explicitly specify a sixth-grade reading level and conduct rigorous manual verification after generation to improve the readability of information [4,22,78,80]. Given that primary health care settings have a better understanding of patients' health literacy and digital preferences, they should play an important role in the development and evaluation process of PEMs, proactively enhance patients' digital literacy, and guide them to participate in the design [37,81,82].

We found that although most studies scored highly on “understandability” in terms of information delivery, they were generally deficient in “actionability,” particularly in the fields of specialized disease diagnostic and therapeutic education and public health and prevention education. This finding highlights a deeper problem: current LLMs are essentially good at retrieving and reorganizing static knowledge, but have difficulty dynamically translating it into health behaviors tailored to specific contexts [28,36,83,84]. This limitation is particularly detrimental in chronic disease management, where the classic knowledge-attitude-practice (KAP) theory emphasizes that patient self-efficacy and behavior change are crucial [85].

This lack of adaptability stems from the model’s lack of contextual awareness (perceiving the patient’s real-life situations), which prevents LLMs from generating personalized PEMs, resulting in information that is theoretically correct but practically ineffective [28,86]. Many studies suggest that materials generated by LLMs should be treated as “drafts” that must be customized based on the patient’s medical history, lifestyle, and living environment to be truly useful [17,87-89]. Engaging patients in the co-design of PEMs can help bridge this gap [81,90-92]. Specifically, through cognitive interviews, user testing, or iterative feedback, patients can identify comprehension and behavioral obstacles, cultural and linguistic preferences based on their own lived contexts, thereby helping PEMs move from being theoretically correct to contextually actionable [90,93].

The high heterogeneity among the included studies hinders the consistency of our conclusions. This review found that the included studies used as many as 12 different readability assessment tools, and half did not use quality assessment tools (such as PEMAT), making it extremely difficult to compare LLMs’ performance across studies and to reach a unified clinical consensus [32,36]. Although LLMs generally outperform social media in terms of reliability, they still cannot match official guidelines in complex clinical decision-making such as surgical management, and carry a risk of information lag [45,46,68,94].

Furthermore, performance disparities between different model versions have raised fairness concerns. If high-quality and readable PEMs require a paid subscription, this could widen the digital divide rather than bridge it, thereby harming the interests of vulnerable groups [95,96]. Much evidence suggests that the most promising application of LLMs in patient education remains the human-machine collaboration model, rather than the deployment of autonomous AI [88,97-99].

Based on this, we propose the following 3 recommendations. First, clinical workflows should be restructured. Physicians should adopt the principle of treating LLMs as “drafting assistants” rather than “final authors,” and involve patients in the design and evaluation of PEMs. Second, promote both standardized evaluation and technical optimization. Establish a unified evaluation framework covering readability, actionability, and accuracy. At the same time, institutions should promote the integration of prompt engineering and retrieval-augmented generation (RAG) technology to address information lag. Finally, prioritize equity and accessibility. Model developers and clinicians should recognize that model performance varies depending on disease complexity. Materials related to chronic diseases may require more rigorous optimization than those for specialized conditions.

This study has several limitations. First, we did not conduct a formal quality assessment of the included studies, and the findings should not be interpreted as conclusions about the strength of specific LLMs or specific clinical scenarios. Second, inclusion was restricted to English-language publications and English-language databases, which may systematically exclude relevant studies from non-English–speaking regions and limit the applicability of our findings in different health care systems and cultural backgrounds. Third, we did not consider grey literature and conference papers. Given the rapid iteration of LLMs and the long delay in the official publication of the literature, this time lag could underestimate the current scope of LLM applications in patient education. Fourth, although we provided the readability range, the conclusion is based on best-case readability, which reflects an upper limit under ideal conditions, and may overestimate the readability of LLMs in the real world. Therefore, the “best potential” cannot be regarded as “conventional performance” and generalized as a general conclusion. Fifth, the rapid evolution of LLMs limits the long-term validity of our conclusions regarding comparisons between specific models [100].

LLMs offer a transformative opportunity to produce effective and high-quality PEMs. However, current applications often fail to meet established health literacy standards, particularly in terms of readability and actionability. To realize their full potential, future efforts must shift from mere capability assessment to optimizing implementation—focusing on prompt engineering, patient participation, human supervision, and retrieval-based accuracy validation. Only through rigorous validation and equitable deployment can LLMs safely bridge the gap between complex medical knowledge and patient understanding, thereby truly empowering patients in their health management.