Enhancing the Readability of Online Patient Education Materials Using Large Language Models: Cross-Sectional Study

Introduction

Online health information is increasingly becoming a leading source for which US adults seek health advice. More individuals use online health information for health advice than family, friends, coworkers, health care professionals, or traditional media [1]. Health information sourced from online materials can supplement the patient-physician conversation regarding health advice [2], however, many individuals without access to a health care provider only seek information online [3]. This is concerning as online materials are not tailored to an individual patient’s specific needs and require readers to discern appropriately between valuable and less relevant health information [4]. Given that 36% of US adults have only basic or below basic health literacy skills [5], navigating the abundant health information online becomes particularly challenging.

Health literacy is an individual’s ability to understand and use health information to make informed decisions [6]. The inability to interpret health information can have serious consequences for patients, as individuals with low health literacy rates often experience poorer health outcomes [7]. They may also be more likely to be readmitted to the emergency department or access inappropriate health services than those with adequate health literacy, contributing to rising health care costs [8,9]. There is a growing recognition that the creators of health information must present materials in a simplified format that encourages understanding, as making patient education materials (PEMs) more readable and understandable has been shown to lead to better comprehension of health-related information among those with low health literacy [6,10]. Several validated measures of readability exist, including the Flesch Reading Ease [11] (FRE), Flesch-Kincaid Grade Level [12] (FKGL), Gunning Fog Index [13] (GFI), and the Simple Measure of Gobbledygook Index [14] (SMOGI). In addition, understandability can be measured by the Patient Education Materials Assessment Tool-Understandability [15] (PEMAT-U) scores.

The National Institutes of Health (NIH) recommends that PEM be written at a maximum reading level of sixth grade [16]. However, PEMs provided by hospitals and those available online are consistently above a sixth-grade reading level [17-19]. Further, one study reviewing 100 PEMs from 3 distinct PEM content vendors providing materials to the National Library of Medicine and electronic health records vendors found most of the PEMs were above an eighth-grade reading level [20]. A meta-analysis of 7891 websites using 13 different readability scales found mean grade level scores ranged from grade 10-15 [21]. Disease-specific studies focusing on stroke and heart failure found materials are consistently above the recommended reading level [22-24], which is especially concerning as they are among the leading causes of death in the United States [25]. While improvements in readability scores have been made over time [17] and human intervention can assist in improving readability [19], challenges remain in bringing readability to an appropriate level on a mass scale.

Large language models (LLMs) can consume massive amounts of data and perform a wide array of tasks that have made them useful tools to understand and generate text [26], making them an enticing option to improve the readability of PEMs. One study evaluated the impact of institutionally developed PEMs using LLMs and found that readability could be improved, but still did not meet the NIH recommended reading level [27]. In another study seeking to improve patient accessibility of published discharge summaries, LLMs enhanced readability and reliably reached the targeted reading level. However, the study concluded that LLM outputs required clinician oversight to ensure accuracy [28].

There are multiple publicly accessible LLMs available for use today, including ChatGPT (OpenAI) [29], Gemini (Google) [30], and Claude (Anthropic PBC) [31]. Potentially, patients or PEM creators could use these resources to simplify online PEM material, but little is known about the efficacy or safety of this approach, though risks of bias and transparency have been identified [32]. Existing literature on LLM simplification often focuses on institution-specific or topic-specific materials [33-35], which limits the broader applicability of the findings. These studies typically address readability improvements within a narrow context, leaving a gap in a holistic understanding of the impact on readability, understandability, and accuracy [27,33,36]. There is a need for research that evaluates PEMs across a wider range of health topics and sources to provide more generalizable insights. As such, the primary objective of this study was to evaluate if LLMs, when prompted, could simplify readability of online PEM content across a range of health topics and sources, while maintaining accuracy and understandability.

Methods

Overview

This cross-sectional study looked at online accessible PEMs accessed between July 1, 2024, and August 30, 2024. All study procedures complied with institutional ethical standards and those set by the Declaration of Helsinki and are reported using both the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology; Multimedia Appendix 1) [37] reporting guidelines for cross-sectional studies and the TRIPOD-LLM (Transparent Reporting of a multivariable model for Individual Prognosis or Diagnosis–LLMs) checklist for reporting studies involving the use of LLMs (Multimedia Appendix 2) [38].

Recruitment

The study used 60 randomly selected PEMs available online from 3 websites. The number 60 was chosen a priori based on feasibility. We selected 3 websites with publicly available health education materials: the American Heart Association [39] (AHA), the American Cancer Society [40] (ACS), and the American Stroke Association [41] (ASA). We selected these 3 organizations as heart disease, cancer, and stroke are among the leading causes of death in the United States [25]. Materials for review were selected from the organization’s website. A total of 20 articles from each site were randomly selected for review.

Intervention

We chose to use free, publicly available generative artificial intelligence (AI) platforms to improve the readability of the 60 PEMs, and we accessed the platforms through the publicly available chat interfaces. We chose free platforms as they help to support equitable access to the technology for all patients; for this reason, we intentionally accessed the platforms through the same web-based interface that patients experience. These platforms were OpenAI’s ChatGPT, Google’s Gemini, and Anthropic’s Claude. Each of the 60 PEMs was entered into the freely available LLMs ChatGPT (Version: GPT-4, released on May 13, 2024, OpenAI), Gemini (Version: Gemini-1.5-flash, released on May 24, 2024, Google), and Claude (Version: Claude 3.5 Sonnet, released on June 20, 2024, Anthropic).

We asked each LLM via prompt to “Translate to a fifth-grade reading level” and pasted the original article text into the LLM. We selected this prompt to model research showing that AI tools improved readability by targeting a lower level than the sixth-grade level recommended by the NIH. This approach, which has been proven successful, was designed to account for variability in grade-level interpretations by LLMs [27]. We then saved copies of each output from the LLM, resulting in 4 total versions of each material (the original plus 3 LLM versions).

Measures

The primary outcome of readability of each PEM and the LLM-simplified PEM was measured by online accessible scoring of FRE, FKGL, GFI, and SMOGI [42]. The PEMAT-U score was manually scored by a project team member using the PEMAT guide [15]. We also extracted the number of words for each PEM and each LLM-simplified PEM. We completed assessments for errors or inaccuracies by reviewing the LLM-simplified PEM and comparing it to the original PEM as the criterion standard. A total of 2 project team members in nonclinical roles completed these reviews independently. If either team member marked the LLM-simplified PEM as having errors or inaccuracies, a physician member of the project team reviewed the article to determine the final outcome. Errors or inaccuracies were defined as instances in which the original message was changed, or the LLM simplified content misrepresented the meaning of the original content which could lead to a different interpretation by the reader.

Statistical Analysis

Descriptive statistics presented include means and SD of the FRE, FKGL, GFI, SMOGI, and PEMAT-U scores, as well as the number of words to measure material length independently of the score. We used a 1-way ANOVA test to test for significant differences in the original PEM of the material between websites. To assess statistically significant differences in readability scores and word count from the original PEM to the LLM-simplified PEM, we used Wilcoxon signed rank test to compare each LLM-simplified PEM to its corresponding original PEM. All data were prepared and analyzed in IBM SPSS (version 28.0.1.1) [43].

Ethical Considerations

The project does not involve human participants in the research and does not require institutional review.

Results

We evaluated 60 unique PEMs from 3 different websites (Table S1 in Multimedia Appendix 3). Mean readability scores across the 3 websites were not significantly different from each other, regardless of the readability used. The AHA had the highest FKGL, GFI, and SMOGI reading scores, and the lowest FRE score, though none were significantly different. The ACS had significantly more words than both the AHA and ASA.

All 3 LLMs had significantly improved readability scores when compared with the original source. Gemini and Claude had greater improvements in readability than ChatGPT for each website. Though improved, there were only 2 instances in which the project target of achieving a mean sixth-grade reading level was met (Chat GPT’s AHA FKGL score and Gemini’s ACA FKGL and GFI score and ASA GFI score). There were 5 instances in which the LLM prompt of achieving a mean fifth-grade reading level was met (ChatGPT’s ASA FKGL score, Gemini’s AHA GFI score, Claude’s AHA GFI score, and Claude’s ACS and ASA GFI score). In 5 other instances, the mean reading level was improved beyond a fifth-grade reading level (Gemini’s AHA and ASA FKGL score, and Claude’s AHA, ACS, and ASA FKGL score). Readability scores and SD for each website’s original PEM and the 3 LLM-simplified PEM are shown in Table 1.

We calculated the average of the 3 grade-level readability scores (FKGL, GFI, and SMOGI) for each PEM version of the material, and compared them across sites. Compared with the original website, ChatGPT improved the readability to 7.6 from 10.1 (P<.001); Gemini, to 6.6 (P<.001); and Claude, to 5.6 (P<.001). Of LLM-simplified versions of the PEM, only Claude consistently achieved a mean fifth-grade reading level on all 3 websites (5.5-5.9), while Gemini did on one of the websites, and ChatGPT on none. Figure 1 shows mean readability scores by website and PEM version.

‎

We assessed accuracy for all LLM-simplified PEM content based on its original version. ChatGPT had no inaccuracies on all 3 sites, while Gemini experienced 2 inaccuracies on the AHA website and Claude experienced one inaccuracy on the ACA and one on the ASA website. Accuracy data is presented in Table 2. The 4 inaccuracies identified are presented in Table 3, with the original and LLM-simplified text.

Discussion

Principal Findings

Our study reveals that the readability of PEMs from the AHA, ACS, and ASA websites significantly exceeds the NIH-recommended sixth-grade reading level. This finding aligns with existing literature, which consistently reports that online PEMs are often written at a higher reading level [21-24]. The mean readability scores for the original PEMs were 10.7, 10, and 9.6 for AHA, ACS, and ASA, respectively.

The application of LLMs such as ChatGPT, Gemini, and Claude demonstrated a substantial improvement in the readability of these materials. Still, despite being prompted to translate the text to a fifth-grade reading level, the LLMs often produced content that was above this target. ChatGPT had the highest mean reading level (7.2-8.3), followed by Gemini (5.7-7.9). Only Claude consistently achieved a mean fifth-grade reading level (5.5-5.9), which is below the NIH recommended maximum level. These results suggest that while LLMs can enhance readability, achieving the desired reading level remains challenging. Alternative prompting techniques that include more description or seek a lower reading level score may yield different results. Although no specific patterns related to content or type of PEM were identified that scored higher or lower, FKGL readability scores were consistently lower than GFI or SMOGI scores, highlighting the variation of the readability scoring algorithms.

Understandability, as measured by the PEMAT-U, saw some significant improvements, though this was not consistent across all sites and LLMs. Some notable improvements within the PEMAT-U by the LLMs focused on word choice by only using medical terms to familiarize the audience and subsequently defining them, organization by breaking the material into short sections if the original material was not, and layout by adding bullet points.

Interestingly, our study found that Google’s Gemini outperformed ChatGPT, contrary to previous research where ChatGPT was more effective than Google’s Bard [27]. This variability is not unexpected, as newer models can exhibit different performance levels, but it does highlight the need for further research to identify the most reliable and effective LLM-based approaches for optimizing PEM readability.

Accuracy is another critical factor when simplifying PEMs. Our study found high accuracy rates for LLM-simplified PEMs, with ChatGPT achieving 100% accuracy and Gemini and Claude at 96.7%. These rates are significantly higher than those reported in previous work on simplified discharge summaries, which had a 54% accuracy rate [28]. The higher accuracy in this study may be attributed to the simpler nature of PEMs compared with the more technically worded discharge summaries. However, the tradeoff between simplification and accuracy remains a concern in this study as well, as Gemini and Claude, the models that performed best at simplification, were the least accurate.

Beyond accuracy concerns, the use of AI in health care presents some inherent risks. One meta-analysis found bias and transparency are 2 primary risks associated with AI in health care [32]. This is an important consideration when using LLMs to simplify PEMs, as important content intended to minimize bias in PEM may be omitted for the sake of simplification. In addition, LLMs are not obviously transparent about the source of data used in their text generation, presenting risks of misinformation and the need for clinical oversight.

Overall, this study demonstrates that while publicly available LLMs hold great promise, they may not yet be fully reliable for direct use by patients to simplify PEMs. However, based on our results, PEM creators who are content experts in their respective fields could certainly start to use LLMs to improve the readability of their own materials. This could be accomplished before the publication of materials online through prompt engineering and human review of the simplified PEM. To gain efficiency, content creators could prompt LLMs to create PEM rather than simplify it and add guardrails to ensure necessary content is maintained. This approach could help address the persistent issue of PEMs being written above the recommended reading level, which disproportionately affects individuals with low health literacy. Given the increasing reliance on online information as a primary source of health information, the adoption of LLMs by online PEM creators has great potential for enhancing patient understanding and improving health outcomes.

Recommendations

Future research should explore more advanced LLM engineering techniques, such as few-shot learning, where LLMs learn through examples included within the prompt to achieve targeted readability goals [28]. One could envision a library of prompts made available through a publicly accessible application designed specifically for this purpose. In addition, integrating LLM self-evaluation, where the model checks its own output for inaccuracies, and generating knowledge prompting, which involves asking the model to generate its knowledge about a topic before providing an answer to reduce hallucinations and improve accuracy, could further enhance the effectiveness of these models. While our study focused on free, publicly accessible LLMs to ensure equitable access, more sophisticated pay-for-play models like OpenAI’s o1 model and Claude Pro may offer greater reliability and effectiveness. In addition, LLMs trained specifically on medical data or fine-tuned for this specific task are promising avenues for future study.

While the out-of-the-box use of LLMs is not yet fully reliable for direct patient use, the significant opportunity for improved performance suggests that this capability is well within reach. With continued advancements and refinements, we are confident that LLM simplification of PEMs can become a safe and effective patient-facing tool that empowers patients to better understand their health conditions and ultimately improves health literacy and outcomes.

Limitations

This study has several limitations. First, it only assessed content from 3 websites, and results may vary with other sources of PEM, especially PEM on rare diseases, where online PEM may be less abundant. Second, the study included only English-language PEMs; materials in other languages may yield different outcomes. Third, we used publicly available content and prompted the LLMs to translate the content to a specific reading level. Different methods, such as asking LLMs to create content from scratch, may produce different results. In addition, the manual calculation of PEMAT-U scores may introduce subjective bias, and the focus on readability and understandability of text ignores the capability of LLMs to create diagrams, infographics, and figures that can help with patient understanding. Further, team members calculating the PEMAT-U scores and accuracy were not blinded to the LLM they were evaluating due to limited project resources, introducing reviewer bias. Finally, while the content was generally accurate, we did not evaluate whether important omissions were made during simplification. Future studies should assess the impact of content loss on the PEMs’ educational value. Although independent reviews of the materials were completed by nonclinical team members to prevent medical knowledge bias in the evaluation of PEMAT-U and accuracy, it’s possible reviews by patients or patient caregivers would respond differently.

Conclusions

Online PEMs consistently exceed the NIH-recommended maximum sixth-grade reading level, posing a challenge for individuals with low health literacy. LLMs offer a promising solution for simplifying PEMs to improve accessibility. Our study demonstrates that LLMs can significantly enhance readability while maintaining high accuracy. However, achieving the desired reading level remains challenging, and human oversight is necessary to ensure the completeness and accuracy of simplified content. Further research is needed to explore advanced LLM techniques and models specifically trained for medical content to optimize the readability and understandability of PEMs.