Outputs from the 2 model conditions were evaluated using a structured scoring framework anchored to the vignette-specific expert reference standard. For each case, scoring focused on whether the response correctly identified the dominant metabolic pattern, prioritized recommendations consistent with guideline-based care, translated the findings into practical patient-facing advice, and avoided clinically inappropriate suggestions.

Three independent reviewers, blinded to study arm allocation, evaluated all outputs using the same predefined rubric. To reduce the likelihood of recognition bias, outputs were deidentified and scored according to content rather than response source. The evaluation framework was applied to both the structured-input experiment and the patient query–style robustness experiment. The detailed scoring rubric is summarized in Table 2, and the final reviewer-scoring dataset is provided in Multimedia Appendix 5.

Three ordinal domains were assessed: metabolic specificity, guideline adherence, and actionability. Each domain was scored on a 5-point Likert scale, with higher scores indicating better performance. For each output, scores from the 3 reviewers were averaged to generate a composite score for each ordinal domain.

Metabolic specificity reflected the extent to which the response was tailored to the metabolic abnormalities and the clinical scenario presented in the case. Higher scores were assigned when the output correctly prioritized the dominant stone-risk pattern and linked specific abnormalities, such as hypercalciuria, hypocitraturia, hyperoxaluria, low urine volume, or urine pH abnormalities, to appropriate dietary recommendations.

Guideline adherence was evaluated by the degree to which the recommendations were consistent with the expert reference standard and current guideline principles. Higher scores were awarded when the response captured the key recommended measures without introducing advice that conflicted with established dietary management strategies for recurrent stone prevention.

Actionability assessed whether responses translated the metabolic interpretation into clear, feasible, and patient-usable dietary guidance. Responses received higher scores when they provided specific instructions, practical priorities, and language that could reasonably support clinical counseling or patient follow-up.

In addition to these ordinal domains, safety was assessed as a binary outcome. A response was classified as unsafe if it omitted or contradicted an important case-specific safety constraint, particularly in scenarios in which otherwise standard dietary advice required modification because of comorbid disease, pregnancy, advanced age, infection-related stones, or other clinically relevant conditions. For the primary safety analysis, consensus failure was defined a priori as a fail judgment assigned by at least 2 of the 3 reviewers.

The primary unit of analysis was the clinical vignette. For the structured-input experiment, paired comparisons were performed between StoneAgent and the standard LLM for each vignette. For the patient query–style robustness experiment, query A and query B represented 2 phrasings of the same underlying case rather than independent observations; therefore, reviewer-averaged scores for the 2 query variants were first aggregated at the vignette level before paired comparison.

For the ordinal domains of metabolic specificity, guideline adherence, and actionability, scores from the 3 independent reviewers were averaged to obtain a composite score for each output. These ordinal outcomes are summarized as medians with IQRs, and means are also reported for descriptive completeness where appropriate. Paired comparisons between StoneAgent and the standard LLM were performed using the Wilcoxon signed-rank test.

Safety was analyzed as a paired binary outcome. For each output, consensus failure was defined a priori as a fail judgment assigned by at least 2 of the 3 reviewers. Paired comparisons of safety pass rates between study arms were evaluated using the exact McNemar test.

Query-level results are presented descriptively to illustrate within-case consistency across the 2 patient-style phrasings, whereas vignette-level aggregation was used for the primary robustness comparison. Interrater reliability for the ordinal rubric domains was assessed using the intraclass correlation coefficient (ICC), based on a 2-way mixed-effects model for absolute agreement [18]. Because reviewer scores were averaged to generate composite vignette-level scores, average-measures ICCs were used for the primary interpretation, with single-measure ICCs calculated for reference. Agreement for the binary safety assessment was summarized, where reviewer-level data were available, using percent agreement and Fleiss κ. All tests were 2-sided, and P<.05 was considered statistically significant. Given the exploratory nature of this in silico comparative study, secondary domain-level comparisons were interpreted as supportive rather than strictly confirmatory. All analyses were performed using SPSS, version 29.0 (IBM Corp).

This study was reported with consideration of the relevant recommendations for evaluating AI-based clinical interaction and decision support approaches. The completed checklist is provided as a supplementary file.

A total of 30 synthetic clinical vignettes were evaluated in the structured-input experiment, and each vignette was also reformulated into 2 patient query–style variants for the robustness experiment. Across both phases, StoneAgent generally outperformed the standard LLM in metabolic specificity, guideline adherence, and actionability, while also demonstrating a more favorable safety profile. The full prompt and response sets are provided in Multimedia Appendices 3 and 4, and the reviewer scoring dataset is provided in Multimedia Appendix 5.

In the structured-input experiment, StoneAgent showed consistently higher paired ratings than the standard LLM across all 3 ordinal domains. Median scores reached 5.00 across metabolic specificity, guideline adherence, and actionability, whereas the standard LLM showed lower and more variable performance. Safety pass rates were numerically higher for StoneAgent (30/30 vs 25/30), although this difference did not reach statistical significance in exact paired testing.

In the patient query–style experiment, the absolute differences between frameworks were more conservative after vignette-level aggregation of query A and query B, but the directional advantage of StoneAgent was retained across the 3 ordinal domains, with numerically higher safety pass rates as well. Taken together, these findings indicate that the observed advantage of StoneAgent was not limited to idealized structured inputs and remained detectable when the same cases were presented in more natural conversational language.

Interrater reliability across the 3 ordinal rubric domains was excellent in both experimental phases. In the structured-input experiment, the average-measures ICC(A,3) was 0.991 for metabolic specificity, 0.961 for guideline adherence, and 0.967 for actionability. In the patient query–style robustness experiment, after vignette-level aggregation of query A and query B, ICC(A,3) values were 0.978 for metabolic specificity, 0.988 for guideline adherence, and 0.974 for actionability. These findings support the use of reviewer-averaged composite scores for vignette-level comparison.

For the binary safety assessment, reviewer-level ratings were available in the query-level dataset, where agreement was also high (Fleiss κ=0.944; overall agreement 99.4%). For the structured-input experiment, only the final consensus safety classification was retained in the exported appendix workbook.

A summary of the paired case-level results for the structured-input experiment is presented in Table 3.

StoneAgent showed a numerically more favorable safety profile than the standard LLM across both experimental phases (Tables 3 and 4). In the structured-input experiment, StoneAgent produced no consensus-fail responses, whereas 5 of 30 responses in the standard LLM group were classified as unsafe or insufficiently constrained (30/30 vs 25/30; exact McNemar P=.06). In the patient query–style robustness experiment, the same directional pattern persisted after vignette-level aggregation (30/30 vs 27/30; exact McNemar P=.25). These findings suggest a potential safety advantage associated with the StoneAgent framework, although the paired comparisons for safety did not reach conventional statistical significance in this sample.

The safety difference was most evident in vignettes in which standard stone-prevention recommendations required explicit modification because of comorbid conditions or competing clinical priorities. These included cases involving advanced chronic kidney disease, heart failure, pregnancy, primary hyperparathyroidism, infection-related stones, and older adults with functional constraints affecting hydration counseling.

Representative cases illustrated how framework-level differences translated into reviewer scoring. In case C09, an infection-related stone scenario, StoneAgent more consistently prioritized infection-focused management and stone clearance before routine dietary prevention advice, whereas the standard LLM was more likely to default to generic recurrence-prevention counseling [6,7,17]. In case C22, a chronic kidney disease scenario, StoneAgent more often modified otherwise standard recommendations related to fluid intake, alkalinization, or electrolyte-related advice in light of renal safety considerations, whereas the standard LLM was more prone to provide broadly reasonable but insufficiently constrained recommendations [6,7,19,20]. These examples highlight that the observed differences were not limited to response detail alone but also involved prioritization of the dominant clinical problem and preservation of case-specific safety logic.

This study evaluated whether a guideline-integrated, safety-aware clinical interaction framework could improve the quality of personalized dietary counseling outputs for recurrent urolithiasis when implemented on the same underlying LLM as the standard LLM interaction condition. In the structured-input experiment, StoneAgent generated recommendations that were more metabolically specific, more closely aligned with guideline-based management, and more actionable than those produced by the standard LLM configuration. Importantly, this performance advantage was not confined to standardized vignette inputs. When the same cases were reformulated as patient query–style prompts, StoneAgent retained a directional advantage across the main evaluation domains after vignette-level aggregation.

The observed difference was not simply a matter of producing longer or more detailed answers. Rather, StoneAgent more consistently identified the dominant metabolic problem, prioritized the most relevant preventive strategy, and modified standard dietary advice when case-specific safety constraints were present. The contrast with the standard LLM was most evident in vignettes involving comorbidity-sensitive counseling, mixed metabolic patterns, or scenarios in which broadly reasonable stone advice could still be clinically misprioritized.

Taken together, these findings support the view that specialty counseling tasks may benefit not only from general language competence but also from explicit clinical framing, guideline grounding, and safety-oriented response scaffolding.

It is important to interpret guideline adherence in the context of the framework design. Because explicit guideline integration was a predefined component of the StoneAgent framework, this outcome reflects the degree to which the framework successfully operationalized guideline-based counseling principles rather than representing an isolated measure of general reasoning ability. Therefore, the comparison evaluates the clinical use of a structured, guideline-oriented interaction approach relative to an unguided LLM interaction.

Prior studies on LLM use in urology have largely focused on general health information, patient education, or broad question-answering tasks [15,16,21-23]. By contrast, the present study addressed a narrower and more clinically constrained task: translating 24-hour urine metabolic findings into personalized dietary recommendations for recurrent stone prevention. This distinction is important because the quality of the response in this setting depends not only on fluency or factual recall but also on correctly prioritizing metabolic abnormalities, preserving guideline-consistent management logic, and respecting case-specific safety constraints.

Our findings also extend the literature by evaluating performance under both structured vignette inputs and patient query–style inputs. This dual design allowed us to assess not only whether a framework could perform well under standardized conditions but also whether its advantage was retained when the same cases were expressed in more natural conversational language. In that sense, the study evaluates whether a clinically structured, guideline-integrated interaction framework can improve task reliability across different input formats when compared with a standard LLM interaction.

More broadly, as general-purpose foundation models continue to improve, the key question may shift from whether they can respond to specialized counseling tasks at all to how they should be structured and governed for reliable use. Our findings suggest that explicit clinical framing and safety-aware scaffolding may still add value even when a general-purpose model is already capable of producing broadly plausible responses [12,14,15].

These findings have potential implications for metabolic stone counseling, particularly in settings where clinicians must convert laboratory data into patient-facing advice under time constraints. A framework such as StoneAgent may support dietary counseling by organizing metabolic interpretation into more consistent and practical recommendations. Its potential value may be greatest not in replacing specialist judgment but in assisting with the translation of metabolic findings into structured counseling points that are easier to communicate during follow-up visits or preventive care discussions.

From an implementation perspective, the potential value of such a framework will depend not only on clinical performance but also on development, maintenance, and integration costs. This study did not perform a health economic evaluation and therefore cannot determine whether framework-assisted counseling would reduce health care expenditures. However, recurrent urolithiasis is associated with substantial downstream costs related to repeated procedures, emergency visits, and long-term management. Future studies should evaluate the cost-effectiveness of guideline-integrated LLM-based counseling tools by considering implementation costs alongside potential reductions in preventable recurrence-related health care service use.

However, higher actionability scores should not necessarily be interpreted as requiring increasingly detailed or prescriptive recommendations. In dietary counseling for recurrent urolithiasis, clinically useful advice must balance specificity with appropriate individualization and safety considerations. Particularly in cases involving comorbidities or competing clinical priorities, a cautious and appropriately constrained recommendation may be preferable to a more extensive but potentially unsuitable dietary plan.

At the same time, the results should not be interpreted as support for fully autonomous dietary management. The outputs evaluated in this study were generated in a simulated setting and should be viewed as decision support candidates rather than stand-alone clinical advice.

Safety is a particularly important dimension in dietary counseling for recurrent urolithiasis because recommendations that appear broadly reasonable in general stone prevention may become inappropriate when important clinical modifiers are present, such as chronic kidney disease, pregnancy, infection-related stones, advanced age, or cardiovascular comorbidity. In this context, the apparent advantage of StoneAgent is less about generating longer or more detailed responses and more about preserving case-specific clinical constraints when translating metabolic findings into advice. Although the safety comparisons in this study did not reach conventional statistical significance, the directional pattern across both experimental phases suggests that explicit safety-oriented scaffolding may be valuable in specialty counseling tasks where otherwise standard recommendations require contextual modification [6,7,19,20,24-26].

The potential risk of unguided LLM use should also be considered. Although general-purpose LLMs can provide broadly appropriate health information, responses generated without clinical constraints may fail to prioritize the dominant metabolic abnormality, provide overly generalized dietary advice, or recommend modifications that are inappropriate for patients with specific comorbidities. In recurrent stone disease, such limitations could theoretically contribute to ineffective prevention strategies, unnecessary dietary restriction, or delayed recognition of conditions requiring specialist evaluation. Therefore, future clinical applications of LLMs in specialty counseling should incorporate explicit safety frameworks, domain-specific validation, and appropriate human oversight.

Several limitations should be considered. First, this study was based on synthetic clinical vignettes rather than real patient encounters. Although the cases were designed to reflect clinically plausible and guideline-relevant metabolic stone scenarios, they cannot capture the full heterogeneity, ambiguity, and communication variability of real-world practice. Second, the patient query–style inputs were derived from the structured vignettes rather than authored by patients themselves and therefore represent a pragmatic approximation of real consultation language rather than a true external validation set. Third, the study was designed as an exploratory in silico benchmarking study using purposively constructed cases to maximize coverage of guideline-relevant and safety-sensitive scenarios, rather than as a power-calculated confirmatory trial. Fourth, the study compared 2 interaction frameworks implemented through the same publicly available web interface of the underlying LLM. Because that interface does not provide API-level control over generation parameters or fixed model snapshot identifiers, the findings should be interpreted as an operational comparative evaluation under real-world usage conditions rather than as a fully parameter-controlled benchmark. Fifth, the study focused specifically on dietary counseling and did not evaluate broader management decisions such as pharmacologic prevention, imaging follow-up, or procedural planning. Finally, response quality was assessed against expert-derived standards rather than real patient outcomes, and the study evaluated isolated response generation rather than longitudinal patient-model interactions. Therefore, the findings do not establish clinical effectiveness, adherence, recurrence reduction, conversational adaptation, or long-term clinical integration.

Additionally, because the intervention being evaluated was a structured prompting framework, the observed performance improvement may partly reflect the contribution of prompt design and task-specific scaffolding rather than the language model alone. Future studies using controlled prompt-ablation designs could help clarify the relative contribution of individual framework components.

Future work should evaluate this type of framework using real patient-authored queries, external clinician reviewers, and prospective clinical workflows. Future evaluations should also examine interactive use cases rather than single-turn responses alone, including how guideline-integrated frameworks respond to additional patient questions, clarification requests, newly available laboratory information, or changes in clinical context over time. Future evaluations should also include patients or simulated cases with multiple concurrent comorbidities to determine whether the framework can appropriately reconcile competing or interacting safety constraints when generating dietary recommendations. Future development could also distinguish between clinician-facing and patient-facing implementations of guideline-integrated LLM frameworks. A clinician-facing version could support interpretation of metabolic findings and preparation of individualized counseling plans within clinical workflows, with subsequent evaluation against patient outcomes. A patient-facing version could support longitudinal reinforcement of clinician-provided recommendations, respond to follow-up questions, and adapt counseling as symptoms, laboratory findings, or clinical circumstances change between formal health care encounters. These complementary use cases warrant separate evaluation of usability, safety, and clinical effectiveness. Qualitative studies involving patients and clinicians, such as interviews or focus groups with individuals who have experience with recurrent stone disease, may provide important insights into usability, trust, communication quality, and barriers to adoption. Comparative evaluation across newer foundation models and alternative guideline-based frameworks would also help clarify which components of the observed benefit are attributable to framework design and which may diminish as baseline model performance improves.

In this in silico comparative study, a guideline-integrated, safety-aware clinical interaction framework generated higher-quality dietary recommendations for recurrent urolithiasis than a standard general-purpose LLM interaction when both were implemented on the same underlying model. This advantage was retained, although in a more conservative form, when the same cases were presented as patient query–style inputs. These findings support the value of explicit clinical framing, guideline grounding, and safety-oriented scaffolding for specialty counseling tasks involving metabolic stone prevention. Further evaluation is needed using real patient-authored queries, external reviewers, and prospective clinical settings.