This multicenter, randomized, parallel-group, double-blind evaluation was conducted in Shanghai. Medical professionals and pediatric patient family members were enrolled and randomly assigned to 1 of 3 groups to evaluate the quality of health education materials from different sources via questionnaire (Figure 1). Participants did not receive any clinical, therapeutic, preventive, or behavioral intervention intended to modify a health-related outcome, and no biomedical, clinical, or patient health outcomes were assessed. The study was registered with the China Clinical Trial Registry under an observational study classification and was retrospectively registered on November 9, 2025 (ChiCTR2500112000). The study design, randomized allocation scheme, evaluation instrument, and primary outcome were determined before participant enrollment and before examination of the study results. No outcome was added, removed, or reclassified on the basis of the observed results.

This study was approved by the Ethics Committee of Shanghai Children’s Medical Center, Shanghai Jiao Tong University School of Medicine (SCMCIRB-K2024201-1) before participant recruitment began. Written informed consent was obtained from all participants.

Participants were recruited between December 2024 and August 2025. A purposive sampling strategy was used to ensure representation across diverse health care settings. Health care professionals were recruited from 2 tertiary hospitals, 1 secondary hospital, and 5 community health service centers. Family members of patients were recruited from 1 tertiary hospital and 1 secondary hospital.

Eligibility criteria for medical professionals included: (1) meeting professional requirements, (2) willingness to participate, and (3) ability to independently complete the questionnaire. Eligibility criteria for pediatric patient family members included (1) being the primary caregiver of a child with asthma, (2) voluntary participation, (3) ability to complete the questionnaire independently, and (4) no self-reported history of conflicts with health care providers.

Data collection was conducted by 9 trained and briefed medical students from Shanghai Jiao Tong University School of Medicine.

Both investigators and participants remained blinded to the source of the materials throughout the study. To ensure data quality, 2 attention-check questions were embedded in the electronic questionnaire. A final question assessed whether participants perceived the materials as LLM-generated. All materials were standardized in font and format to prevent identification based on textual characteristics.

All source questions and educational responses used in this study were presented in simplified Chinese. The PEARs were sourced from the validated Chinese Children’s Asthma Action Plan: 100 Questions and Answers [16]. All AI-generated materials were produced using GPT-4o (OpenAI) [17] through the ChatGPT web interface between August 13 and August 29, 2024. Materials were generated using the default interface settings without additional parameter adjustments by the researchers. For each question, the prompt was submitted once to generate one AI-generated response (AI-GR). Using prompt engineering strategies based on established methodologies [18], simplification instructions were applied to create AI-simplified responses (AI-SRs), aiming to enhance readability while preserving core meaning (Figure 2). Subsequently, each simplification prompt was submitted once to generate the corresponding AI-SR. No multiple candidate responses were generated, compared, or selectively retained. The complete prompts and detailed generation workflow are provided in Multimedia Appendix 1.

A pediatric pulmonologist with >10 years of experience screened all generated content using a 5-point Likert scale (Multimedia Appendix 2). In total, 87 items (excluding 13 problematic ones) scoring ≥2 points were retained for the final item banks. Additionally, GPT-4o generated AI-GRs with a total response time of 357 seconds and AI-SRs with a total response time of 341 seconds.

Participants were randomly assigned in a 1:1:1 ratio to 1 of 3 parallel material groups: PEARs, AI-GRs, and AI-SRs. The 1:1:1 ratio referred to participant-level assignment across the 3 groups; each participant evaluated only the type of material corresponding to the assigned group. Within each parallel group, each participant was presented with 5 question-response pairs randomly selected from the corresponding material pool (via a random number generator). After reviewing the 5 pairs, each participant completed the 19-item questionnaire based on the information adoption model once to provide an overall evaluation of the material set and contributed one total scale score. Randomization was used solely to determine the material bank evaluated by each participant; no clinical or health-related intervention intended to modify a health outcome was administered.

The primary outcome was the total score of a 19-item, 5-point Likert scale (1=strongly disagree to 5=strongly agree) based on the information adoption model, assessing 4 dimensions: information quality [19,20], credibility [21-23], usefulness [24,25], and adoption intention [26]. The scale demonstrated good internal consistency (Cronbach α >0.8 for total and subscale scores) and satisfactory construct validity (all factor loadings >0.5).

Secondary outcomes included individual dimension scores (quality: 4 items; credibility: 6 items; usefulness: 6 items; adoption intention: 3 items). Linguistic features (total words, sentences, difficult words, and mean sentence length) were analyzed using the Chinese Readability Index Explorer [27-30]. Each participant evaluated the source of the selected assessment materials at the end of the questionnaire (whether authored by medical professionals or generated by other tools). Demographic data (age, gender, education, work experience, residence, income, etc) were collected.

Sample size was calculated using G*Power (version 3.1.9.2) based on a 3-group comparison of total scores. With an effect size Cohen d=0.5, α=.025 (2-sided), and power (1–β)=0.80, 192 participants were required. Accounting for 20% attrition, 240 participants (80 per group) were targeted.

Scale scores were analyzed using general linear models including material source (PEARs, AI-GRs, and AI-SRs), participant population (medical professionals and pediatric patient families), and their interaction. F statistics, df, P values, and partial η² were reported for each model effect. On the basis of estimated marginal means, prespecified comparisons of AI-GRs versus PEARs and AI-SRs versus PEARs were conducted within each participant population. The 4 prespecified comparisons for the total scale score were adjusted collectively using the Holm procedure. Model-estimated mean differences, 95% CIs, unadjusted P values, and Holm-adjusted P values were reported. Statistical significance was defined as a 2-sided Holm-adjusted P value of <.05. The same models were used for secondary analyses of the 4 questionnaire dimensions. The 16 comparisons arising from the 2 participant populations, 4 dimensions, and 2 prespecified comparisons per dimension were treated as a separate family and adjusted using the Holm procedure. Results were interpreted according to the adjusted P values. Subgroup analyses were restricted to the total score and were considered exploratory. Demographic data were summarized using medians (IQRs) for continuous variables and percentages for categorical variables. Linguistic measures were summarized as medians and IQRs and compared across the 3 matched versions using Friedman tests, followed by paired Wilcoxon signed-rank tests. The 12 pairwise comparisons across the 4 linguistic measures were collectively adjusted using the Holm method.

Among 267 approached medical professionals and 330 patients’ family members, 243 (91%) professionals (PEARs: n=79, 32.5%; AI-GRs: n=81, 33.3%; AI-SRs: n=83, 34.2%) and 276 (83.6%) patients’ family members (PEARs: n=96, 34.8%; AI-GRs: n=90, 32.6%; AI-SRs: n=90, 32.6%) completed the evaluation and passed attention checks.

The median age of medical professionals was 37 (range 30‐46) years. Most were women (194/243, 79.8%), and highly educated (231/243, 95% held bachelor’s degrees or higher). More than half (129/243, 53.1%) had 10 or more years of experience, with 40.7% (99/243) pediatricians, 36.2% (88/243) general practitioners, and 23% (56/243) pediatric nurses. Patients’ family members were primarily parents (265/276, 96%), with a median age of 38 (range 35‐41) years, and 71.4% (197/276) were women. Most (235/276, 85.6%) of them had a bachelor’s degree or higher, 77.2% (213/276) resided in urban areas, and 58.3% (161/276) reported monthly household incomes exceeding ¥15,000 (¥1=US $0.15 as of August 17, 2026). The children under their care were predominantly boys (177/276, 64.1%), with a mean age of 8.1 years; 81.2% (224/276) were receiving medication, and 59.8% (165/276) had asthma for more than 1 year. Nearly all participants (233/243, 95.9% professionals; 262/276, 94.9% caregivers) had prior experience using electronic devices (Table 1).

In the general linear model for the overall scores, the main effect of material source was statistically significant (F_2,513=3.330; P=.04; partial η²=0.013), and the effect size was small. A statistically significant main effect of participant population was also observed (F_1,513=35.984; P<.001; partial η²=0.066), indicating a difference in overall scores between medical professionals and caregivers. The material source-by-participant population interaction was not statistically significant (F_2,513=0.116; P=.89; partial η²<0.001), providing no statistical evidence that differences among the material sources varied according to participant population (Table 2).

Among medical professionals, mean total evaluation scores were 72.97 (SD 14.88) for PEARs, 76.17 (SD 12.59) for AI-GRs, and 72.98 (SD 12.64) for AI-SRs. The model-estimated mean difference was 3.20 for AI-GRs (95% CI −0.99 to 7.39; unadjusted P=.13; Holm-adjusted P=.53) and 0.01 for AI-SRs (95% CI −4.18 to 4.20; unadjusted P>.99; Holm-adjusted P>.99). Corresponding mean scores among pediatric patient family were 64.18 (SD 15.76), 68.77 (SD 16.47), and 65.47 (SD 16.52). AI-GRs had a numerically higher score than PEARs, with a model-estimated mean difference of 4.59 (95% CI 0.28-8.90) and an unadjusted P value of .037. However, this difference was no longer statistically significant after Holm adjustment across the 4 predefined total score comparisons (adjusted P=.15). The model-estimated mean difference between AI-SRs and PEARs was 1.29 (95% CI −3.03 to 5.61; unadjusted P=.56; Holm-adjusted P>.99; Table 3).

General linear model analyses showed significant main effects of participant population across all 4 dimensions, with medical professionals generally providing higher ratings than pediatric patient families (all P<.001). Material source had significant overall main effects on the information usefulness and information adoption dimensions (both P=.03), but not on the information quality or information credibility dimensions. No significant interactions between material source and participant population were observed for any dimension, indicating that the effects of material source did not differ significantly between the two participant populations (Table 2).

In the prespecified comparisons, the only nominally significant unadjusted difference was observed for information usefulness among pediatric patient families, with AI-GRs receiving higher scores than PEARs (mean difference 1.93, 95% CI 0.26-3.60; P=.02). However, this difference did not remain statistically significant after Holm adjustment across the 16 dimension-specific comparisons (adjusted P=.38). None of the remaining comparisons were statistically significant either before or after adjustment. Thus, the adjusted analyses provided no evidence of significant differences between AI-GRs or AI-SRs and PEARs on any individual scale dimension (Table 3).

Subgroup analyses demonstrated generally balanced overall ratings across demographic strata (Figure 3). Although point estimates suggested potentially higher ratings among respiratory specialty nurses, no significant interaction effects were detected across any subgroups in either population (medical professionals: gender, P=.97; education, P=.42; age, P>.99; digital technology use experience, P=.75; profession, P=.42; and work years, P=.71; pediatric patient families: gender, P=.87; education, P=.89; age, P=.95; digital technology use experience, P=.52; residence, P=.33; and income, P=.83).

Only 32% (166/519) of all participants correctly identified the material sources. While 80.6% (141/175) of participants in the PEARs group accurately recognized published expert-authored content, merely 7.3% (25/344) in the AI-generated groups correctly identified the content as LLM-generated. Medical professionals (79/243, 32.5%) and patient’s family members (88/276, 31.9%) demonstrated similar correct identification rates (Figure 4).

Compared with AI-GRs, AI-SRs had fewer words (median 128, IQR 106‐203 vs median 157, IQR 116‐335), fewer difficult words (median 52, IQR 42‐75 vs median 63, IQR 45‐126), and a shorter average sentence length (median 9.33, IQR 8.53‐10.05 vs median 10.07, IQR 9.28‐10.82; all Holm-adjusted P<.001). Compared with PEARs, AI-SRs had more sentences (median 19, IQR 14‐24 vs median 16, IQR 13‐19; Holm-adjusted P<.001) and a shorter average sentence length (median 9.33, IQR 8.53‐10.05 vs median 10.24, IQR 9.52‐11.50; Holm-adjusted P<.001). Between AI-GRs and PEARs, only the difference in sentence count remained significant after Holm adjustment, with AI-GRs having fewer sentences (median 14, IQR 11‐33 vs median 16, IQR 13‐19; Holm-adjusted P=.01). No other pairwise differences remained significant after Holm adjustment (Figure 5).

This multicenter, randomized, double-blind evaluation study compared Chinese PEARs with AI-GRs and AI-SRs across health care professionals and caregivers. The primary outcome captured participants’ perceived quality, credibility, usefulness, adoption, and overall scores through the information adoption model. AI-GRs received numerically higher overall scores than PEARs in both participant populations, whereas AI-SRs and PEARs had similar numerical scores. After Holm correction, none of the prespecified comparisons between either AI group and PEARs was statistically significant for the overall score or its 4 dimensions. The results showed no statistically detectable differences in the planned comparisons, while the numerical patterns provide a basis for more targeted investigation.

The favorable reception of AI-GRs accords with earlier studies in which LLM responses to asthma and rheumatology education materials were positively evaluated [31,32], as well as with broader evidence identifying patient education as a growing use of general-purpose LLMs [33]. By incorporating blinded assessments from both medical professionals and pediatric patient family members and comparing the materials with an established expert-authored resource, this study extends previous work beyond expert scoring to include the perspectives of intended users.

The modest numerical advantage of AI-GRs may be related to fluency, organization, and perceived accessibility, which can shape readers’ impressions of health information, although these attributes were not separately measured in this study [34-38]. The nominal difference in perceived usefulness among pediatric patient family members was also attenuated after correction across the 16 dimension-specific comparisons (adjusted P=.38), suggesting that this pattern should be interpreted cautiously. Overall, the findings are most informative about perceived credibility, relevance, usefulness, and willingness to use the materials. They provide preliminary support for the acceptability of LLM-assisted health education, while objective accuracy, clinical safety, comprehension, and behavioral effects remain complementary domains for purpose-designed evaluation [39].

The AI-SR findings further suggest that linguistic simplification is not captured fully by reductions in word count, difficult words, or sentence length. Although all 3 indices decreased, the scores on the overall scale and 4 dimensions did not increase. Similar work has shown that LLMs can lower the estimated reading level of health information but has also emphasized the importance of outcomes beyond readability scores and continued human oversight [40]. A ceiling effect offers one plausible explanation for the present pattern. When ratings cluster near the upper end of a scale, the limited remaining response range makes additional improvement more difficult to detect [41]. The pediatric patient family sample was predominantly highly educated, urban, and relatively affluent and may therefore have found the original materials readily accessible; if initial acceptance was already high, ratings concentrated near the upper end of the scale may have left limited room for simplification to produce further improvement. At the same time, shortening language may remove contextual detail or weaken coherence, both of which contribute to text difficulty but are rarely incorporated into Chinese readability assessment systems [42-44]. Future studies should examine more diverse health literacy and socioeconomic backgrounds and evaluate readability together with comprehension, factual retention, and the ability to apply information in realistic tasks.

Participants’ judgments of authorship add a further dimension to the interpretation of perceived credibility. Although overall source identification accuracy was 32%, responses were markedly asymmetric: 80.6% (141/175) of participants assigned to PEARs selected a human source, whereas only 7.3% (25/344) of those assigned to the two LLM groups selected an LLM source. This pattern may be consistent with a directional tendency to attribute polished health information to a human expert. Fluent, credible presentation may shape trust before the underlying accuracy and safety have been independently evaluated. Work on transparency and explainability similarly distinguishes subjective trust from appropriately calibrated reliance on an AI system [45]. In patient education, the observed acceptability therefore points most plausibly toward an LLM-assisted workflow in which generated or simplified materials undergo qualified professional review, objective accuracy and safety assessment, clear provenance disclosure, and monitoring tied to the model version and prompt process [46-49].

Several limitations should be considered. PEARs were taken from a polished, committee-validated public resource, whereas the LLM materials were generated for this study and prescreened; these production differences may have influenced the evaluations. The prescreening was conducted by a single pediatric pulmonologist, which may have introduced reviewer-dependent judgment into content selection. Because the source resource was publicly available, possible prior model exposure to it or related material cannot be excluded. The predominantly highly educated, urban, and relatively affluent family sample may limit generalizability to populations with lower health literacy or fewer socioeconomic resources, while the asymmetric source judgments may partly reflect directional response anchoring. The use of 1 GPT-4o version, 1 language, and 1 prompt strategy further limits generalizability to other models and workflows. Finally, although the study received ethics approval before participant recruitment and the study design and outcomes were specified before enrollment, the registration was not formally completed and publicly posted until after participant recruitment had ended. The study was therefore retrospectively registered. Nevertheless, no outcome was added, removed, or reclassified on the basis of the observed results. Future randomized evaluation studies will be registered, where applicable, before enrollment of the first participant.

This randomized, double-blind study examined medical professionals’ and caregivers’ perceptions of LLM-generated, LLM-simplified, and published expert-authored pediatric asthma education responses. After Holm correction, the prespecified analyses did not identify statistically significant differences in overall information adoption model scores or the 4 information adoption model dimensions. Linguistic analyses indicated that AI-based simplification reduced word count, difficult-word count, and average sentence length. These findings provide preliminary evidence regarding the use of generative AI for developing and simplifying pediatric asthma education materials. Further research is needed to assess their clinical accuracy, accessibility, real-world effectiveness, and applicability across languages, populations, and health conditions.