Hypospadias, a common congenital external genital malformation in male children, exhibits a globally increasing incidence [1]. Surgery remains the sole treatment option, yet postoperative complications such as urinary fistula and urethral stricture can directly affect reproductive function and quality of life in adulthood [2,3]，placing considerable psychological burdens and information needs on caregivers [4,5]. Current health care resources often fall short of providing timely, personalized guidance, leading caregivers to seek information online—where verifying the authenticity of available content presents a persistent challenge [6,7]. The emergence of generative large language models (LLMs) offers a new pathway to mitigate this information asymmetry. Models such as GPT-4 produce medical reasoning and conversational language at near-human fluency [8], and earlier work has shown promise in empathetic oncology consultations and accurate ophthalmic advice [9,10]. However, hypospadias sits at the intersection of pediatric complexity and reproductive privacy, where fluent but incorrect advice can be especially harmful [11-13]. Identifying which LLMs can answer caregivers’ perioperative questions safely and clearly—and understanding which evaluation dimensions matter most to clinicians versus families— therefore has direct consequences for patient education and equity of access.

Prior LLM-evaluation work in adjacent settings (eg, bone health queries [14], emergency care patient questions [11], ophthalmology [10], and oncology [9]) has typically addressed knowledge-benchmarking or single-perspective accuracy questions. While Faraj et al [15] recently evaluated LLM performance on pediatric hypospadias knowledge using a structured Campbell-Walsh multiple-choice questionnaire , their study was restricted to professional-perspective multiple-choice benchmarking. The broader stakeholder needs and evaluation-methodology question addressed here—multidimensional, dual-perspective (clinician and caregiver) evaluation of open-ended clinical consultation responses for a sensitive pediatric condition—has not been previously characterized.

While research indicates that performance variations and risks of fabricated or unverifiable citations exist across different models [14], and that highly fluent text may mask underlying errors or even dangerous advice [11-13], existing evaluation evidence predominantly relies on English-language contexts. Topics such as pediatric reproductive potential carry distinct cultural sensitivities and taboos within Chinese society. Validating the applicability of Chinese LLMs in this scenario is therefore an urgent priority, yet direct evidence regarding the performance of Chinese models such as DeepSeek in professional medical consultations remains limited [16].

This study differs from prior work in 3 respects. First, instead of single-perspective benchmarking or multiple-choice testing [15], we use a dual-perspective (clinician and lay caregiver) evaluation of open-ended consultation responses. Second, the evaluation is stratified by clinical risk level, so sensitive perioperative scenarios are scored separately from routine logistical ones. Third, we run parallel, independent audits of bibliographic citation authenticity and of clinical content safety. Together these 3 components provide a safety-and-utility profile that, to our knowledge, has not previously been reported for pediatric urology artificial intelligence (AI).

Building on evaluation framework of Sezgin et al [17], this study assesses 5 representative LLMs—ChatGPT-4o, Gemini-2.5-Pro, Zhipu Qingyan, DeepSeek, and OpenEvidence—in the perioperative consultation context for pediatric hypospadias. The primary aim is to evaluate these models on three components: (1) a double-blind, randomized-label, and forced-ranking comparison from caregiver and expert perspectives; (2) independent citation authenticity verification using a 5-category V/PV/F/G/NR scheme (V: Verifiable, PV: Partially Verifiable, F: Fabricated, G: Guideline-Based, Nonspecific, and NR: No References); and (3) a European Association of Urology (EAU) 2025–anchored clinical safety audit. As a secondary aim, ranking robustness is tested by poststratification-weighted sensitivity analyses.

We prespecified 2 confirmatory hypotheses. First, LLM performance varies across both clinical correctness and patient-centered communication quality, and no single model is uniformly superior on every dimension. Second, stakeholder ratings, citation verifiability, and clinical safety are dissociable: high stakeholder ratings or accurate bibliographic citations do not in themselves guarantee clinical safety. Three additional exploratory aims are (1) whether caregivers, given unrestricted choice, prioritize higher-risk clinical questions; (2) whether model ratings vary systematically across sociodemographic strata or disease severity; and (3) whether expert and caregiver evaluations diverge at the dimension level.

Each hypothesis maps directly to a design element. The first hypothesis is tested by the double-blind forced-ranking format, which removes central-tendency bias, together with the separate expert and caregiver rubrics that decouple clinical correctness from communication quality. The second hypothesis is tested by reading the stakeholder ranking, the citation verification, and the safety audit side by side rather than collapsing them into a single composite score. The exploratory aims are addressed through stratified subgroup analyses and free-choice question selection, kept separate from the confirmatory tests.

The study adopted a 2-cohort design for caregiver recruitment: cohort A (question-bank screening, n=34) and cohort B (LLM response evaluation, n=36), with no overlapping participants between the 2 cohorts. A total of 23 experts were enrolled for professional evaluation. Major demographic characteristics collected for caregivers included age, sex, relationship to the patient, education level, monthly household income, and employment status. Patient (child) characteristics included age and hypospadias type. For the expert panel, demographic and professional characteristics collected included age, clinical experience (years), and professional seniority title. Standardized classifications were applied: monthly household income was categorized using the 2024 per-capita disposable-income data for urban residents from the National Bureau of Statistics of China [18] (low ≤3500 RMB/US $≤488; medium 3501‐5400 RMB/US $488 to US $752; and high >5400 RMB/US $>752; 7.18 RMB=US $1 as of April 1, 2025, source: People's Bank of China), and hypospadias types were classified as distal, midshaft, proximal, or severe, which were merged into Distal/Midshaft (Type I and II) and Proximal/Severe (Type III and IV) for subgroup analyses. Expert professional title was categorized into junior, intermediate, and senior titles.

All participants were recruited via convenience sampling at the Department of Pediatric Urology of a tertiary hospital in Chengdu, China. Data collection, including LLM response generation and evaluator recruitment, was completed in April 2025. This manuscript adheres to the Chatbot Assessment Reporting Tool guidelines [19] (Checklist 1) and the APA Journal Article Reporting Standards for Quantitative Research.

All eligible participants were informed of the study purpose, procedures, and potential risks before enrollment. All participants signed written informed consent voluntarily. No monetary or in-kind compensation was provided to participants. Recruitment of caregivers followed a sequential, nonoverlapping design: cohort A completed the question-bank screening survey; cohort B completed the evaluation. Institutional review board approval and ethical review standards are detailed in Ethical Considerations section.

No a priori sample size calculation or power analysis was performed, as the study size was determined by convenience and feasibility within a single specialized tertiary pediatric urology department. Precision of parameter estimates was quantified post hoc using 95% CIs calculated for median scores (exact order-statistic method) and effect-size metrics (rank-biserial correlation and Spearman ρ). The robustness and generalizability of the achieved sample size against convenience-sampling imbalance were evaluated using a poststratification-weighted sensitivity analysis.

Eight covariates were included: caregiver education, family income, employment status, hypospadias type, expert professional seniority, expert age, expert clinical experience, and clinical risk level of consultation questions.

Caregiver age, relationship to the patient, and patient age were collected for descriptive purposes and not included in formal stratified analysis.

All procedures complied with the Declaration of Helsinki and were approved by the Biomedical Ethics Review Committee of West China Hospital, Sichuan University (approval number 2025 Review [2457]). All expert and caregiver participants provided written informed consent prior to participation, including consent for anonymized data analysis and publication. This noninterventional, cross-sectional design evaluated AI-generated text rather than assessing human health outcomes; prospective clinical trial registration was not mandated by International Committee of Medical Journal Editors guidelines. Caregiver demographic data were collected anonymously and stored in a password-protected database accessible only to the research team, and no real patient data or identifiable health information was entered into any LLM at any point—all queries were based on standardized scenarios from the preconstructed question bank. No monetary, in-kind, or other form of compensation was provided to participants in this study. No images of individual participants, patients, or users are included in the manuscript or any supplementary material.

The flow of participants across recruitment, eligibility, allocation, and analysis is shown in Figure 1. Caregivers were recruited in 2 sequential phases using independent, nonoverlapping cohorts. For cohort A (question-bank screening), 40 caregivers were assessed for eligibility, of whom 6 (15.0%) were excluded—3 did not meet inclusion criteria and 3 declined to participate—leaving 34 (85.0%) enrolled who all completed the 40-question survey. For cohort B (LLM response evaluation, independent of cohort A), 40 caregivers were likewise assessed for eligibility, of whom 4 (10.0%) were excluded—2 were unable to provide complete data and 2 had children with complex comorbidities—leaving 36 (90.0%) enrolled who all completed the double-blind forced-ranking evaluation across the 5 models, 10 questions, and 4 dimensions. The expert panel was recruited via professional referral from the Department of Pediatric Urology and the affiliated nursing department; of 27 invited specialists, 23 (85.2%) completed the full evaluation, while 4 (14.8%) were unable to complete on schedule, primarily due to clinical workload constraints (with no exclusions for conflicts of interest). No data were excluded from analysis from any stream.

The period of participant recruitment and repeated data collection occurred in April 2025. We recruited 2 independent caregiver cohorts and a panel of clinical experts sequentially. All eligible participants were screened and enrolled following the predefined inclusion and exclusion criteria, and no participants dropped out after formal enrollment.

This study evaluated 5 mainstream LLMs—Gemini-2.5-Pro, DeepSeek, ChatGPT-4o, Zhipu Qingyan, and OpenEvidence—for perioperative consultation in pediatric hypospadias. The first confirmatory hypothesis—that LLM performance would vary across clinical correctness and patient-centered communication, with no model proving uniformly superior—was supported. Gemini-2.5-Pro produced the most comprehensive answers in both expert and caregiver evaluations; DeepSeek ranked second, with caregivers giving particularly high marks on Empathy; and OpenEvidence finished last from both perspectives despite producing the highest verifiable citation rate. No model led on every dimension. The principal ranking remained stable across 8 sociodemographic strata, across the 3 clinical risk levels, and across the poststratification-weighted sensitivity analysis (Spearman ρ=1.00 between weighted and unweighted rankings), arguing against demographic or clinical confounding as drivers of the observed pattern.

The second confirmatory hypothesis—that stakeholder ratings, citation verifiability, and clinical safety are dissociable dimensions of model behavior—was also supported. The 3 pillars yielded inconsistent rankings of the same models. OpenEvidence had the highest citation accuracy (V=100%) and yet accumulated 5 of 9 (56%) panel-wide Severe events, showing that accurate citations do not, on their own, guarantee clinically safe content; whether they are necessary cannot be answered from cross-sectional data of this design. DeepSeek illustrated the inverse: a citation fabrication rate of 33% co-occurred with zero Severe events. Gemini-2.5-Pro, also free of fabricated citations (F=0%), likewise recorded none. Fabricated citations and unsafe clinical content therefore appear to be distinct failure modes that require different mitigation strategies.

The 3 exploratory aims, reported with caution given uncontrolled family-wise error rates, each yielded a substantive finding. When caregivers were given unrestricted choice, they disproportionately selected higher-risk clinical questions: High- and medium-risk items accounted for 60% of the final top 10 despite contributing only 20% of the original pool, suggesting that caregivers naturally weight high-stakes clinical information above routine logistical queries, consistent with the heightened concern documented among caregivers of children with serious conditions [5], as well as their active information-seeking behaviors [6] and raising the bar for accuracy and safety in any AI model used in real-world patient education. Subgroup analysis showed a monotonic decline in OpenEvidence ratings as caregiver income rose (P trend =.033), with a marginal trend in the opposite direction for Gemini-2.5-Pro; higher-income caregivers, who may have greater health literacy [33,34] or stronger expectations about delivery style, appear less tolerant of OpenEvidence’s academic phrasing. On the Empathy dimension, caregivers rated DeepSeek and Zhipu Qingyan higher than ChatGPT-4o, although the underlying mechanisms—linguistic familiarity, cultural framing, or affective resonance—were not measured directly here and remain hypotheses for future work.

Most prior work on medical AI has focused on model accuracy and treated accuracy as a proxy for usefulness [10]. Our sociodemographic data complicate that picture. The monotonic decline in OpenEvidence ratings with rising income, alongside the marginal rising trend for Gemini-2.5-Pro in the same group, mirrors earlier observations [10,11] that specialized information presented without lay-audience adaptation can hinder patient decision-making [35]. The caregiver qualitative comments point in the same direction: level of detail and comprehensibility were the 2 most frequent reasons for ranking. Deploying AI tools without sufficient attention to comprehensibility risks widening, rather than narrowing, disparities in access to understandable medical information [33,34,36]—a health-equity issue that must be explicitly addressed in all AI deployment plans.

On the affective dimensions, the locally adapted models DeepSeek and Zhipu Qingyan were rated higher on Empathy by caregivers than ChatGPT-4o. This is consistent with Masoud et al [37], who showed that language models tuned to specific linguistic and cultural contexts more closely reflect local social norms. ChatGPT-4o’s pattern was characteristic: experts gave it moderate Quality scores, but caregivers rated it poorly on Empathy and Addressing Concerns. Mesko and Topol [38] have argued that the strict safety alignment training of general-purpose models can push them toward formulaic disclaimers that read as cold; our caregiver ratings are compatible with that account. The trade-off is real, however: DeepSeek’s higher Empathy was paired with the highest citation-fabrication rate; so the Chinese-adapted models in our panel have not yet reconciled empathy with citation reliability.

The dissociation between citation accuracy and clinical safety extends earlier concerns. As demonstrated in health information overload research, highly technical evidence presented without clinical context can function as “information noise” that impairs patient comprehension, and Athaluri et al [12] and Alkaissi and McFarlane [13] documented the misinformation risk posed by fabricated citations that are superficially credible. Our V/PV/F/G/NR consensus classification adds resolution: DeepSeek and Zhipu Qingyan together account for the majority of fabricated citations, while Gemini-2.5-Pro’s guideline-leaning strategy prevents fabrication at the cost of bibliographic granularity. Within the disease-severity dimension, DeepSeek’s statistically significant performance drop in Proximal or Severe (types III and IV) cases (r=0.336, moderate effect) suggests that probabilistic models still have trouble matching the case-specific judgment human experts apply in complex hypospadias surgery—consistent with Topol’s [39] view (and GPT-4 analysis by Lee et al [40]) that human and machine intelligence work best in combination, not substitution. A practical translation is the tiered human-machine workflow proposed by Liyanage et al [41] and Krajcer [42]: validated models handle low-risk educational queries, while severe deformities, complication management, and complex surgical planning stay with human specialists. This is in line with LLM evaluation work in urolithiasis [43] and cornea care [44], where comprehensibility and triage gaps were similarly identified.

Several sources of bias and threats to internal validity should be kept in mind. To counter the central tendency bias inherent in traditional absolute rating scales [45], we used a double-blind forced-ranking design with per-question randomized labels. This design forced evaluators to make distinct comparative judgments, effectively curbing “satisficing”—the tendency to default to neutral scores when cognitive demands are high [46]. As a result, the internal validity of our between-model comparisons was strengthened. Sampling was nevertheless by convenience at a single tertiary center in Chengdu, China, and the expert and caregiver groups likely underrepresent primary care, rural, and other cultural settings; with mothers making up 28 of 36 caregivers (77.8%), the Empathy and communication-quality findings cannot be assumed to extend to fathers or to other family structures.

Evaluator seniority represents a key potential source of assessment bias. Stratified analysis revealed that senior specialists rated Gemini-2.5-Pro monotonically higher and OpenEvidence monotonically lower as seniority increased, likely reflecting a higher sensitivity to citation-fabrication risks or stricter clinical standards. This aligns with recent evidence from Faraj et al [15], who found that senior, certified pediatric urologists exhibit significantly sharper clinical reasoning in hypospadiology assessments than junior peers. Consequently, accounting for evaluator seniority is essential, as long-term clinical maturity dictates a more rigorous, risk-aware standard when auditing AI-generated counseling content.

Regarding imprecision of measurement protocols, constructs such as eHealth literacy, cultural identity, linguistic preference, and state anxiety were not measured using validated psychometric instruments (eg, the eHealth Literacy Scale [eHEALS] [36] and the State-Trait Anxiety Inventory [STAI] [47]).

The overall number of tests and overlap among tests merit consideration. The principal ranking proved robust across 8 sociodemographic strata, 3 clinical risk levels, and poststratification weighting, providing strong evidence that the findings are not artifacts of multiple testing. However, subgroup analyses and exploratory free-choice question findings should be interpreted as hypothesis-generating rather than confirmatory. The adequacy of sample sizes and sampling validity is supported by the same robustness checks (ie, stratification and weighting), but the single-site, single-disease design still limits generalizability beyond this cohort.

Several contextual factors limit how far these results travel. All 5 models were tested at one moment in time (April 6, 2025) on free-tier web interfaces with no exposed version identifiers, so the ranking values are explicitly time-anchored [21,48]; the evaluation method itself, and the clinician- and caregiver-prioritized dimensions identified here, should remain useful for benchmarking later model versions.

The clinical scope is also narrow. We tested perioperative consultations for a single pediatric malformation, so the model-specific conclusions need replication before they are applied to other settings; the evaluation structure (forced-ranking + citation audit + EAU-anchored safety audit) can be reused across other pediatric or adult specialties, as analogous multidimensional expert evaluations have proven viable in both adult urology and cross-disciplinary domains [43,44]. However, the specific model rankings cannot.

Finally, the 8-reviewer safety audit assessed static written outputs, not real-time clinical interactions. A clinical deployment would need prospective patient outcome monitoring, formal red-flag detection, and institutional safety review [38,49], together with cross-cultural validation [50] to test whether the dimension priorities (technical accuracy vs empathy) and the caregiver expert divergence pattern hold elsewhere. The evaluation framework, anchored by the V/PV/F/G/NR 5-category classification with dual-reviewer adjudication (Cohen κ=0.702, substantial agreement) and canonical-source reverification of all 30 disagreements, is portable across clinical domains; the specific results are not.

These implications rest on several methodological strengths: a dual-perspective design capturing complementary clinician- and caregiver-rated dimensions; double-blind forced-ranking with randomized labels to mitigate central-tendency bias [45,46]; a 5-category consensus citation classification (V/PV/F/G/NR) with dual-reviewer adjudication (Cohen κ=0.702) and canonical-source reverification; an 8-reviewer EAU 2025–anchored clinical safety audit; robust cross-stratum ranking stability confirmed across sociodemographic subgroups, clinical risk tiers, and poststratification weighting (Spearman ρ=1.00); and a time-stamped, configuration-specific protocol (Multimedia Appendix 1) that bounds temporal validity and establishes a reproducible, unoptimized baseline.

This unoptimized baseline provides a systematic starting point for testing future prompt-engineering and retrieval-augmented interventions linked to the specific deficits we identified. Negative-constraint prompting—instructing models to cite only verifiable literature or to declare “insufficient evidence”—could test whether citation fabrication rates can be suppressed without degrading response quality [12,13]. Few-shot prompting [51] with standardized exemplar physician responses could target specific dimension-level deficits, such as ChatGPT-4o’s low Empathy scores or OpenEvidence’s poor Comprehensibility, provided these trials are conducted prospectively with concurrent evaluator recruitment to avoid recall bias. Retrieval-augmented generation (RAG) grounded in authoritative clinical resources (eg, EAU/AUA guidelines) is the most clinically actionable next step [52]; the specific errors flagged in our safety audit—off-indication testosterone recommendations, erroneous recovery timelines, and missing emergency-triage instructions—are exactly the failure modes RAG architectures are designed to prevent. Future prospective studies evaluating these technical optimizations should also incorporate validated psychometric instruments, such as the eHEALS [36] and the STAI [47], at the time of evaluation to enable direct correlation of these psychological constructs with dimension-level model ratings.

No single LLM was uniformly best for perioperative consultation in pediatric hypospadias. Beyond the specific rankings, which will date as models update, this study contributes 2 findings that should remain useful across model generations: a characterization of which evaluation dimensions clinicians and caregivers each prioritize, and direct evidence that bibliographic accuracy and clinical content safety are dissociable dimensions and need to be assessed separately. The accompanying evaluation framework—dual-perspective forced-ranking, multidimensional rubric, independent citation verification, 8-reviewer guideline-anchored safety audit, and risk-stratified analysis—provides a reproducible baseline against which future prompt-engineering and retrieval-augmented interventions can be tested. For perioperative uses, AI should be deployed under a tiered human-machine collaboration model, with mandatory clinician oversight in high-risk scenarios [41,42].