This cross-sectional study looked at online accessible PEMs accessed between July 1, 2024, and August 30, 2024. All study procedures complied with institutional ethical standards and those set by the Declaration of Helsinki and are reported using both the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology; Multimedia Appendix 1) [37] reporting guidelines for cross-sectional studies and the TRIPOD-LLM (Transparent Reporting of a multivariable model for Individual Prognosis or Diagnosis–LLMs) checklist for reporting studies involving the use of LLMs (Multimedia Appendix 2) [38].

The study used 60 randomly selected PEMs available online from 3 websites. The number 60 was chosen a priori based on feasibility. We selected 3 websites with publicly available health education materials: the American Heart Association [39] (AHA), the American Cancer Society [40] (ACS), and the American Stroke Association [41] (ASA). We selected these 3 organizations as heart disease, cancer, and stroke are among the leading causes of death in the United States [25]. Materials for review were selected from the organization’s website. A total of 20 articles from each site were randomly selected for review.

We chose to use free, publicly available generative artificial intelligence (AI) platforms to improve the readability of the 60 PEMs, and we accessed the platforms through the publicly available chat interfaces. We chose free platforms as they help to support equitable access to the technology for all patients; for this reason, we intentionally accessed the platforms through the same web-based interface that patients experience. These platforms were OpenAI’s ChatGPT, Google’s Gemini, and Anthropic’s Claude. Each of the 60 PEMs was entered into the freely available LLMs ChatGPT (Version: GPT-4, released on May 13, 2024, OpenAI), Gemini (Version: Gemini-1.5-flash, released on May 24, 2024, Google), and Claude (Version: Claude 3.5 Sonnet, released on June 20, 2024, Anthropic).

We asked each LLM via prompt to “Translate to a fifth-grade reading level” and pasted the original article text into the LLM. We selected this prompt to model research showing that AI tools improved readability by targeting a lower level than the sixth-grade level recommended by the NIH. This approach, which has been proven successful, was designed to account for variability in grade-level interpretations by LLMs [27]. We then saved copies of each output from the LLM, resulting in 4 total versions of each material (the original plus 3 LLM versions).

The primary outcome of readability of each PEM and the LLM-simplified PEM was measured by online accessible scoring of FRE, FKGL, GFI, and SMOGI [42]. The PEMAT-U score was manually scored by a project team member using the PEMAT guide [15]. We also extracted the number of words for each PEM and each LLM-simplified PEM. We completed assessments for errors or inaccuracies by reviewing the LLM-simplified PEM and comparing it to the original PEM as the criterion standard. A total of 2 project team members in nonclinical roles completed these reviews independently. If either team member marked the LLM-simplified PEM as having errors or inaccuracies, a physician member of the project team reviewed the article to determine the final outcome. Errors or inaccuracies were defined as instances in which the original message was changed, or the LLM simplified content misrepresented the meaning of the original content which could lead to a different interpretation by the reader.

Descriptive statistics presented include means and SD of the FRE, FKGL, GFI, SMOGI, and PEMAT-U scores, as well as the number of words to measure material length independently of the score. We used a 1-way ANOVA test to test for significant differences in the original PEM of the material between websites. To assess statistically significant differences in readability scores and word count from the original PEM to the LLM-simplified PEM, we used Wilcoxon signed rank test to compare each LLM-simplified PEM to its corresponding original PEM. All data were prepared and analyzed in IBM SPSS (version 28.0.1.1) [43].

The project does not involve human participants in the research and does not require institutional review.

Our study reveals that the readability of PEMs from the AHA, ACS, and ASA websites significantly exceeds the NIH-recommended sixth-grade reading level. This finding aligns with existing literature, which consistently reports that online PEMs are often written at a higher reading level [21-24]. The mean readability scores for the original PEMs were 10.7, 10, and 9.6 for AHA, ACS, and ASA, respectively.

The application of LLMs such as ChatGPT, Gemini, and Claude demonstrated a substantial improvement in the readability of these materials. Still, despite being prompted to translate the text to a fifth-grade reading level, the LLMs often produced content that was above this target. ChatGPT had the highest mean reading level (7.2-8.3), followed by Gemini (5.7-7.9). Only Claude consistently achieved a mean fifth-grade reading level (5.5-5.9), which is below the NIH recommended maximum level. These results suggest that while LLMs can enhance readability, achieving the desired reading level remains challenging. Alternative prompting techniques that include more description or seek a lower reading level score may yield different results. Although no specific patterns related to content or type of PEM were identified that scored higher or lower, FKGL readability scores were consistently lower than GFI or SMOGI scores, highlighting the variation of the readability scoring algorithms.

Understandability, as measured by the PEMAT-U, saw some significant improvements, though this was not consistent across all sites and LLMs. Some notable improvements within the PEMAT-U by the LLMs focused on word choice by only using medical terms to familiarize the audience and subsequently defining them, organization by breaking the material into short sections if the original material was not, and layout by adding bullet points.

Interestingly, our study found that Google’s Gemini outperformed ChatGPT, contrary to previous research where ChatGPT was more effective than Google’s Bard [27]. This variability is not unexpected, as newer models can exhibit different performance levels, but it does highlight the need for further research to identify the most reliable and effective LLM-based approaches for optimizing PEM readability.

Accuracy is another critical factor when simplifying PEMs. Our study found high accuracy rates for LLM-simplified PEMs, with ChatGPT achieving 100% accuracy and Gemini and Claude at 96.7%. These rates are significantly higher than those reported in previous work on simplified discharge summaries, which had a 54% accuracy rate [28]. The higher accuracy in this study may be attributed to the simpler nature of PEMs compared with the more technically worded discharge summaries. However, the tradeoff between simplification and accuracy remains a concern in this study as well, as Gemini and Claude, the models that performed best at simplification, were the least accurate.

Beyond accuracy concerns, the use of AI in health care presents some inherent risks. One meta-analysis found bias and transparency are 2 primary risks associated with AI in health care [32]. This is an important consideration when using LLMs to simplify PEMs, as important content intended to minimize bias in PEM may be omitted for the sake of simplification. In addition, LLMs are not obviously transparent about the source of data used in their text generation, presenting risks of misinformation and the need for clinical oversight.

Overall, this study demonstrates that while publicly available LLMs hold great promise, they may not yet be fully reliable for direct use by patients to simplify PEMs. However, based on our results, PEM creators who are content experts in their respective fields could certainly start to use LLMs to improve the readability of their own materials. This could be accomplished before the publication of materials online through prompt engineering and human review of the simplified PEM. To gain efficiency, content creators could prompt LLMs to create PEM rather than simplify it and add guardrails to ensure necessary content is maintained. This approach could help address the persistent issue of PEMs being written above the recommended reading level, which disproportionately affects individuals with low health literacy. Given the increasing reliance on online information as a primary source of health information, the adoption of LLMs by online PEM creators has great potential for enhancing patient understanding and improving health outcomes.

Future research should explore more advanced LLM engineering techniques, such as few-shot learning, where LLMs learn through examples included within the prompt to achieve targeted readability goals [28]. One could envision a library of prompts made available through a publicly accessible application designed specifically for this purpose. In addition, integrating LLM self-evaluation, where the model checks its own output for inaccuracies, and generating knowledge prompting, which involves asking the model to generate its knowledge about a topic before providing an answer to reduce hallucinations and improve accuracy, could further enhance the effectiveness of these models. While our study focused on free, publicly accessible LLMs to ensure equitable access, more sophisticated pay-for-play models like OpenAI’s o1 model and Claude Pro may offer greater reliability and effectiveness. In addition, LLMs trained specifically on medical data or fine-tuned for this specific task are promising avenues for future study.

While the out-of-the-box use of LLMs is not yet fully reliable for direct patient use, the significant opportunity for improved performance suggests that this capability is well within reach. With continued advancements and refinements, we are confident that LLM simplification of PEMs can become a safe and effective patient-facing tool that empowers patients to better understand their health conditions and ultimately improves health literacy and outcomes.

This study has several limitations. First, it only assessed content from 3 websites, and results may vary with other sources of PEM, especially PEM on rare diseases, where online PEM may be less abundant. Second, the study included only English-language PEMs; materials in other languages may yield different outcomes. Third, we used publicly available content and prompted the LLMs to translate the content to a specific reading level. Different methods, such as asking LLMs to create content from scratch, may produce different results. In addition, the manual calculation of PEMAT-U scores may introduce subjective bias, and the focus on readability and understandability of text ignores the capability of LLMs to create diagrams, infographics, and figures that can help with patient understanding. Further, team members calculating the PEMAT-U scores and accuracy were not blinded to the LLM they were evaluating due to limited project resources, introducing reviewer bias. Finally, while the content was generally accurate, we did not evaluate whether important omissions were made during simplification. Future studies should assess the impact of content loss on the PEMs’ educational value. Although independent reviews of the materials were completed by nonclinical team members to prevent medical knowledge bias in the evaluation of PEMAT-U and accuracy, it’s possible reviews by patients or patient caregivers would respond differently.

Online PEMs consistently exceed the NIH-recommended maximum sixth-grade reading level, posing a challenge for individuals with low health literacy. LLMs offer a promising solution for simplifying PEMs to improve accessibility. Our study demonstrates that LLMs can significantly enhance readability while maintaining high accuracy. However, achieving the desired reading level remains challenging, and human oversight is necessary to ensure the completeness and accuracy of simplified content. Further research is needed to explore advanced LLM techniques and models specifically trained for medical content to optimize the readability and understandability of PEMs.