We followed a multistage systematic workflow to develop the PCaPLMM_SFT (Figure 1). The process included 5 major steps: data preparation, construction of QA datasets with RAG and LLMs, model training, referee model design, and model evaluation. It forms a closed-loop pipeline from knowledge extraction to model optimization and assessment.

The data were primarily obtained from English-language PubMed publications. Publications between February 2015 and February 2025 were retrieved using search terms related to prostate cancer and lifestyle management, including lifestyle, exercise, and diet. The full PubMed search strategy and eligibility criteria are provided in Multimedia Appendix 1. In addition to original research papers, secondary research, such as systematic reviews and meta-analyses, was also included to supplement real-world evidence gaps.

To meet LLMs’ training requirements, all raw documents underwent standardized preprocessing before inclusion. Through multiple rounds of cleaning and semantic extraction, irrelevant content—such as author information, correspondence addresses, DOIs, and copyright statements—was removed. The remaining content was transformed into JSON format suitable for LLM learning (eg, “text”: “paragraph content”). For complex review papers and documents with irregular structures, we applied a hybrid automated manual annotation strategy to maintain semantic consistency and ensure high data quality, providing a reliable foundation for subsequent QA construction.

Cleaned texts were segmented into minimal semantic units, or chunks, based on paragraph boundaries, topic continuity, and medical semantic integrity rather than arbitrary fixed-length splitting. For each chunk, source metadata, including publication information, lifestyle domain, original section, and evidence source, were retained to ensure traceability. Duplicated, noninformative, or structurally abnormal chunks were removed through automated filtering and manual spot-checking. Each chunk was embedded into high-dimensional semantic vectors using the BGE-M3-embedding model and stored in a FAISS (Facebook Artificial Intelligence Similarity Search)–based vector database together with the original text and metadata. This process formed a structured KB enabling efficient and traceable semantic retrieval for downstream QA and inference tasks [15].

During QA generation, query vectors were generated from predefined keywords or semantic seeds. Top-k semantic retrieval (k=20) was performed on the KB to obtain relevant knowledge slices. The value of k=20 was selected as a pragmatic balance between evidence coverage and noise control: a smaller k may miss complementary evidence, whereas a larger k may introduce weakly relevant fragments and distract the model from the most relevant information [16-18]. Retrieved content and preconstructed questions were combined and fed into an LLM via prompting templates, producing answers grounded in medical evidence. RAG was used both for (1) retrieving evidence to generate QA training data and (2) retrieving evidence at inference time to ground responses. Retrieved slices were ranked by similarity, and no additional fixed similarity threshold was applied before response generation. The current system did not implement a formal evidence-grade weighting algorithm or automated conflict adjudication mechanism for inconsistent retrieved slices. When retrieved evidence was heterogeneous, the model was guided by evidence-oriented prompts to generate cautious and individualized responses while avoiding absolute conclusions. This approach was intended to reduce hallucinations and support the medical accuracy and contextual coherence of the outputs.

Based on the prostate cancer lifestyle knowledge slices, core factual questions were extracted through medical templates. These base questions were then reformulated by Qwen3-Turbo into more natural, conversational, and patient-oriented queries. To prevent redundancy and semantic conflicts, semantic similarity filtering was applied to remove duplicate or stylistically inconsistent queries, thereby improving diversity and consistency.

Each patient-style question and its corresponding knowledge slice were combined using a unified prompt template and input into Qwen3-Turbo for answer generation. Because Qwen3-Turbo was used as the primary QA generator, the generated corpus may reflect model-specific biases in linguistic style, knowledge coverage, safety-related phrasing, and reasoning patterns. To reduce unconstrained model generation, answers were generated from retrieved literature-based knowledge slices rather than solely from the model’s parametric knowledge. The prompt required the model to prioritize evidence-based recommendations, precise terminology, and consistency with the retrieved medical evidence. The generated QA pairs were formatted in ShareGPT JSON style to ensure compatibility for SFT.

To ensure data quality, a manual and automated quality control process was applied. We manually reviewed generated QA pairs using a stratified sampling strategy; this manual review was designed as a stratified quality control audit rather than a complete manual validation of the entire corpus. The review was conducted independently by 2 researchers (XW and HH) with medical backgrounds. Manual assessment evaluated factual accuracy, evidence alignment, safety, linguistic clarity, structural validity, and actionability. This audit was used to identify common quality problems, refine editing criteria, and guide subsequent corpus cleaning and standardization. Interrater agreement metrics were calculated and are reported in Multimedia Appendix 2. Discrepant samples were further reviewed and used to guide correction, revision, or exclusion. This process was supplemented by automated checks, including semantic deduplication, terminology filtering, format validation, and logical conflict detection. Together, these procedures removed misleading, ungrammatical, or structurally abnormal samples, and improved the professionalism, naturalness, and standardization of the dataset [19].

To prevent train-test leakage, we adopted a base-question-group splitting strategy. Each base question and its 1‐2 paraphrased variants were grouped as a single unit and assigned entirely to either the training or evaluation set to avoid semantic overlap across datasets. In addition, multilevel deduplication and semantic similarity filtering were applied. All queries were encoded using BGE-M3, and cosine similarity between training and evaluation sets was computed. Cross-set query pairs with high similarity (near-paraphrase cases) were automatically removed, ensuring strict semantic isolation between training and evaluation data.

To clarify the functional scope of the QA workflow, the QA task was defined as a composite generative process performed autonomously by PCaPLMM_SFT after continued pretraining and SFT. The task integrated two capabilities: (1) medical knowledge–based reasoning and (2) generation of patient-oriented lifestyle recommendations. Inputs included a fixed system prompt (eg, “You are PCaPLMM_SFT...provide professional and evidence-based responses”), brief patient information, and a lifestyle-related query. The model first performed medical semantic interpretation and evidence-based reasoning using the structured literature KB and then converted the inferred content into clear, empathetic, and actionable recommendations. Outputs were restricted to a single evidence-grounded and medically accurate answer. Task performance was evaluated across the 5 dimensions: evidence alignment, comprehensibility, relevance, empathy, and feasibility.

This study adopted a 2-stage training strategy for PCaPLMM_SFT. Stage 1 performed parameter-efficient continued pretraining using low-rank adaptation adapters on the Baichuan2-7B-Chat backbone. Training was conducted in LLaMA-Factory with a 2048-token context window, the AdamW optimizer (learning rate 2×10^–⁴), cosine scheduling with 200 warm-up steps, weight decay, gradient clipping (maximum norm=1.0), a per-device batch size of 4, and 8 gradient accumulation steps for 3 epochs. Stage 2 fine-tuned the stage 1 checkpoint using domain-adaptive supervised fine-tuning with low-rank adaptation under a 1024-token context window, AdamW (2×10^–⁵), and a fixed 3-epoch schedule.

The referee models were implemented by accessing the application programming interfaces of DeepSeek-R1 and Qwen3-Max through an open model service platform. Using prompt engineering, we constructed structured, outline-based scoring instructions, forming an LLM-based referee mechanism. This LLM-as-a-judge framework was adopted based on prior studies showing that large LLMs can generate structured evaluation signals under standardized rubrics, although results may be affected by prompt sensitivity and model-specific preferences [20-22]. The evaluation prompt templates incorporated knowledge slices, role definitions, scoring criteria, and explicit restrictions (Multimedia Appendix 3) to support a consistent evaluation context and produce reproducible assessment outputs. To examine the robustness of the referee LLM outputs, we further assessed their interround stability, intermodel consistency, and human-LLM consistency.

The referee mechanism assessed each candidate answer across 5 dimensions—evidence alignment, comprehensibility, relevance, empathy, and feasibility—using the detailed definitions and scoring criteria (Multimedia Appendix 4). A predefined safety rule was also applied to flag responses with potential patient safety risks. Because PCaPLMM_SFT was designed for patient education and lifestyle guidance rather than for diagnosis or treatment decision-making, unsafe outputs were operationalized using a threshold for potential patient harm: responses were considered unsafe if they could reasonably mislead patients toward harmful self-management behaviors or unsupervised changes in medical care. This predefined safety rule was used only for post hoc evaluation and not as a trigger for inserting disclaimers during generation. Unsafe labels were used for analysis rather than exclusion; flagged responses remained in the evaluation denominator for error analysis and reporting of representative failure cases. Responses were labeled “unsafe” when (1) the referee assigned a score of 1 on any subdimension of evidence alignment, indicating inconsistency with established findings; (2) the recommendation carried potential for direct patient harm, such as inappropriate medication suggestions, unsafe or contraindicated lifestyle or exercise advice, or omission of essential safety precautions; or (3) the answer encouraged patients to initiate, discontinue, or modify medical treatment without clinician supervision.

A total of 2500 patient-style queries from English and Chinese test sets covering the 5 lifestyle management scenarios were used for evaluation. Candidate answers from all models were scored independently and blindly by the referee model in 2 rounds. Each dimension was rated on a 5-point Likert scale, ranging from “strongly disagree” (1) to “strongly agree” (5) (Multimedia Appendix 4).

To evaluate the stability of scores at the individual dimension level, each “question× dimension” pair was treated as an independent scoring unit. Intraclass correlation coefficient (ICC) (3,k) was computed using a 2-way mixed-effects model (consistency and average measures). Additionally, Pearson and Spearman correlations were calculated based on a concatenated score vector stacking the 5-dimension–level scores for each question. This approach measures consistency on both continuous and ordinal scales.

To further validate the reliability of the referee LLM evaluations, we conducted a small-scale manual assessment by human experts. Three experts with ≥10 years of experience in prostate cancer management (2 urologists and 1 prostate cancer research specialist) independently evaluated the model-generated responses. A total of 50 QA samples were randomly selected, covering 5 core lifestyle scenarios (10 samples per scenario), based on prior studies [23,24]. This expert assessment served as a supplementary validation component to examine whether automated evaluation signals were broadly aligned with clinical judgment across key dimensions such as evidence alignment, safety, usefulness, feasibility, and patient-centered communication [21,25]. All texts were fully deidentified, and experts received only the QA pairs without any model identifiers to minimize potential evaluation bias. The expert assessment criteria, scoring dimensions, and consistency evaluation methods were kept consistent with those used in the referee LLM evaluation. The sample expert evaluation form for assessing multiple model outputs on lifestyle management in patients with prostate cancer is provided in Multimedia Appendix 5.

All statistical analyses were performed using R (version 4.3.1; R Foundation for Statistical Computing) and Python (version 3.10; Python Software Foundation) with the SciPy (version 1.11; SciPy Community), pandas, and numpy libraries. Referee evaluation scores of candidate models were compared using the Mann-Whitney U test for 2 independent groups. In addition to the U statistic and 2-sided P values, standardized Z scores and effect sizes (r) were calculated, where r=|Z|/N and N is the total sample size. Effect size interpretation followed Cohen thresholds: 0.1=small, 0.3=medium, and 0.5=large. The descriptive statistics (mean, SD) were summarized for each group, along with the corresponding 95% CIs. All multiple comparisons were corrected for the false discovery rate using the Benjamini-Hochberg procedure. A 2-sided P<.05 was considered statistically significant.

This study did not involve patient recruitment, clinical intervention, or the collection or processing of patient-level identifiable information and therefore did not constitute a clinical trial. The study protocol, including the expert evaluation process, was reviewed and approved by the Medical Ethics Committee of Nantong University, with the approval number 2025‐25. The evaluation was conducted by invited domain experts, who were informed of the study purpose, evaluation procedures, and intended use of the results before participation. All experts participated voluntarily and provided informed consent. The evaluation results are reported only in aggregate form, and no personally identifiable information is presented in the manuscript.

Briefly, 7197 records were identified from PubMed, 7195 records were screened, 3188 reports were assessed for eligibility, and 2211 studies were finally included for KB construction. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram of our review is shown in Figure 2.

Based on 2211 included publications, we constructed the PCaPLMM_SFT-Train dataset tailored to prostate cancer lifestyle management scenarios (Table 1). The dataset comprised 1516 original studies and 695 secondary research papers. The content covered 5 core themes: diet and nutrition, physical activity, weight management, psychological support, and medication adherence. After automated cleaning and manual review, irrelevant content and potentially sensitive information were removed. The texts were then structured and labeled by theme to form the foundational corpus.

Within the RAG framework, we used Qwen3-Turbo to automatically generate QA pairs. Questions were produced through template-based extraction, followed by paraphrasing to simulate patient-style queries. Answers were generated from knowledge slices retrieved from the structured KB and were manually screened to ensure accuracy and consistency. To further enhance bilingual understanding and response generation, we constructed a bilingual English-Chinese patient-style QA corpus from the same English-language source evidence through patient-oriented reformulation and RAG-based answer generation. Although non–English literature databases were not independently searched, the generated corpus encompassed a broad range of lifestyle-related domains and incorporated culturally and regionally specific expressions. After 2 rounds of review, we obtained 42,330 single-turn QA pairs and 3008 multiturn dialogues.

The QA generation system integrated structured medical literature knowledge with semantic retrieval and LLM-based generation to produce high-quality, automated patient-style QA content. The KB was constructed by importing unstructured medical texts, segmenting them into semantic vectors, and extracting metadata. A total of more than 150,000 knowledge slices were generated, covering all major lifestyle-related conversational scenarios. Figure 3 illustrates the KB management interface, which includes literature import (Figure 3A), content editing (Figure 3B), structured indexing (Figure 3C), and semantic debugging modules (Figure 3D). Qwen3-Turbo was used to automatically generate patient-style questions, achieving interactive linkage between knowledge construction and data generation.

For each knowledge slice, the system generated 5 base questions aligned with lifestyle contexts. Each question was then paraphrased 1‐2 times by an LLM to enrich the linguistic diversity while preserving its core medical meaning. All questions were documented with their contents, scenario, generation type (original or paraphrased), and source reference. Examples are presented in Table 2.

Following knowledge construction and question generation, we produced QA pairs that were explicitly grounded in medical evidence. For instance (Figure 4A), given the query “Having recently learned that I have prostate cancer, I would like some guidance on diet that may help with lifestyle adjustments,” the system retrieved 7 relevant research fragments (Figure 4B). The generated answer integrated dietary recommendations such as plant-based diets, reduced red meat and sugar intake, and appropriate fat consumption. The outputs were professional, structured, and standardized in ShareGPT JSON format (Figure 4C). To quantify dataset quality, we manually reviewed a stratified sample of 2400 QA pairs, representing 5.62% (2400/42,705) of the prefinal QA candidate corpus used for corpus auditing. Each sample was categorized by required editing level (Multimedia Appendix 6): 50.71% (1217/2400) needed only minor phrasing or structural refinements, 24.58% (590/2400) required major revisions due to factual or evidence misalignment, 22.79% (547/2400) were accepted without changes, and 1.92% (46/2400) were discarded for severe or noncorrectable errors. The major revision rate reflects QA quality and not the residual error in the final SFT dataset. Samples requiring major revision were revised according to retrieved evidence, while noncorrectable or potentially misleading samples were removed. The validated editing criteria derived from this audit were then applied to clean the remaining dataset. Representative failure cases are provided in Multimedia Appendix 7.

The continued pretraining phase consisted of more than 2200 training steps. The loss decreased from approximately 2.9 to below 1.8 and stabilized after approximately 1600 steps (Figure 5A). A total of 23,263,936 tokens were used during this phase. In the subsequent SFT phase, 45,338 structured QA samples were used, with a total training volume of 60,788,864 tokens. The loss declined from approximately 2.0 to approximately 0.9 and converged after approximately 1750 steps (Figure 5B). The final model weights and configuration files have been publicly released on the Hugging Face platform (see the Data Availability section).

Across both evaluation rounds, PCaPLMM_SFT consistently outperformed Baichuan2-7B-Chat across all dimensions and surpassed GPT-3.5-Turbo in most metrics (Table 3). Under Qwen3-Max, PCaPLMM_SFT achieved evidence alignment mean scores of 4.156 (SD 0.591) and 4.155 (SD 0.591), significantly higher than Baichuan2-7B-Chat (|r|=0.601‐0.614; P<.001) and GPT-3.5-Turbo (|r|=0.299‐0.305; P<.001). The model also showed clear advantages in feasibility dimension (mean 3.191, SD 0.717; mean 3.202, SD 0.706), again outperforming both baselines (P<.001).

DeepSeek-R1 produced consistent findings: PCaPLMM_SFT showed significant gains in evidence alignment (mean 4.345, SD 0.501; mean 4.310, SD 0.629; P<.001) and delivered top performance in comprehensibility (4.624‐4.657) and empathy (4.634‐4.637), with significant margins over Baichuan2-7B-Chat (|r|≥0.219; P<.001).

Across the 2 referee LLMs, interrater and interround consistency analyses showed overall moderate to good agreement (Table 4). In round 1, Qwen3-Max and DeepSeek-R1 demonstrated moderate interrater consistency (ICC(3,k)=0.540; Pearson r=0.382; Spearman ρ=0.415), indicating generally comparable scoring patterns despite differences in scoring sensitivity. In round 2, interrater agreement was higher than in the first round (ICC(3,k)=0.747; Pearson r=0.600; Spearman ρ=0.655), reflecting a consistent scoring tendency between the 2 models when independently evaluating the same QA scenarios.

Within-model interround consistency showed clear differences: Qwen3-Max exhibited high reproducibility (ICC(3,k)=0.816; Pearson r=0.690; Spearman ρ=0.719), whereas DeepSeek-R1 demonstrated only moderate stability (ICC(3,k)=0.507; Pearson r=0.343; Spearman ρ=0.378). Overall, although the 2 referee models differed in scoring variability, their scoring trends remained reasonably aligned, supporting the robustness of the comparative evaluation across repeated assessments.

Across 2 independent generations, low-scoring outputs were grouped into 4 error categories (Table 5). PCaPLMM_SFT showed the lowest error rates overall, with evidence misalignment: 3.68%, overly general or insufficiently actionable advice: 9.28%, imprecise or insufficiently patient-centered language: 5.68%, and hallucinations: 5.32%. Baichuan2-7B-Chat demonstrated substantially higher error rates in all categories, particularly evidence misalignment (20.66%) and hallucinations (12.88%). GPT-3.5-Turbo showed intermediate performance, with a notably low rate of patient-centeredness errors (1.42%) (Multimedia Appendix 8). Evidence misalignment and hallucinations were treated as potentially risky error categories because they may mislead patients or provide unsupported lifestyle recommendations, although actual patient harm could not be directly assessed in this offline evaluation without real patient interactions. Overall, PCaPLMM_SFT presented fewer potentially risky errors, suggesting better evidence alignment and reliability in generating lifestyle recommendations.

A total of 3 experts independently evaluated 50 deidentified QA samples, and their ratings were compared with those of the 2 LLM referee models (Table 6). Interexpert agreement was moderate, with ICC(3,k) ranging from 0.332 to 0.431 across expert pairs, and corresponding Pearson correlations of 0.394‐0.471 and Spearman correlations of 0.380‐0.511.

The 2 referee LLMs (Qwen3-Max vs DeepSeek-R1) demonstrated higher intermodel agreement (ICC(3,k)=0.689; Pearson r=0.527; Spearman ρ=0.506), indicating a relatively consistent scoring pattern between models. When comparing human experts with LLM referees, consistency was moderate (ICC(3,k)=0.474, 95% CI 0.410‐0.530), with Pearson r=0.367 and Spearman ρ=0.365. These results suggest that although humans and LLM referees share similar overall ranking tendencies across QA outputs, divergence remains in fine-grained scoring decisions. Human experts’ evaluation scores for the candidate models are provided in Multimedia Appendix 9.

We selected 5 common lifestyle management themes—diet, exercise, weight management, medication adherence, and psychological support—to construct the representative patient-style QA scenarios. These were used to evaluate the adaptability and conversational ability of PCaPLMM_SFT in natural patient queries (Table 7). Table 7 illustrates the model responses across these scenarios and not the best-case outputs. The full evaluation set of 2500 patient-style queries and representative failure cases and error-category analyses are provided in Table 5 and Multimedia Appendix 8.

In the dietary management scenario, the query “How can I adjust my diet to better manage weight and maintain muscle strength while undergoing androgen deprivation therapy (ADT)?” elicited model-generated outputs recommending the consumption of whole grains, fruits, vegetables, lean proteins, and healthy fats, together with calcium and vitamin D supplementation. The model further advised limiting processed foods, sugary beverages, and high-fat snacks, combining dietary changes with resistance training, and underscored the need for individualized planning under physician supervision.

In exercise-related tasks, the model generated recommendations tailored to treatment phase and physical fitness level. For psychological support, the generated responses conveyed empathy and humanistic care. For medication adherence and weight management, the answers incorporated behavioral interventions, reminders, and risk prevention strategies. Overall, the model frequently generated safety-related statements such as “consult your physician” and “tailor interventions to individual assessment.” These statements reflected medical communication patterns learned during SFT and cautious response generation guided by the system prompt, rather than deterministic rule-based safety guardrails. PCaPLMM_SFT consistently outperformed Baichuan2-7B-Chat and demonstrated comparable or superior performance to GPT-3.5-Turbo across all 5 lifestyle scenarios in 2 independent evaluation rounds conducted by Qwen3-Max and DeepSeek-R1 (Table 8).

Under Qwen3-Max evaluation, PCaPLMM_SFT achieved significantly higher scores than Baichuan2-7B-Chat across all scenarios (diet, exercise, psychological support, medication adherence, and weight management; P<.001), with medium-to-large effect sizes (|r| ranging from 0.479 to 0.746). Compared with GPT-3.5-Turbo, PCaPLMM_SFT showed clear advantages in specific scenarios, particularly exercise, psychological support, and weight management (|r| ranging from 0.104 to 0.578), while maintaining comparable performance in others (eg, diet).

Under DeepSeek-R1 evaluation, the superiority of PCaPLMM_SFT over Baichuan2-7B-Chat was consistent and significant across all 5 scenarios, with the largest effect sizes observed in psychological support and medication adherence (P<.001; |r| ranging from 0.515 to 0.841). Comparisons against GPT-3.5-Turbo showed a similar favorable pattern, with PCaPLMM_SFT significantly outperforming the benchmark in most tasks (P<.001; |r| ranging from 0.116 to 0.462).

In this study, we developed and implemented PCaPLMM_SFT, a SFT LLM designed to support healthy lifestyle management for patients with prostate cancer. By integrating a structured literature-based KB, semantic retrieval, and instruction tuning, we generated high-quality QA pairs covering multiple lifestyle-related dialogue scenarios. Furthermore, 2 referee models were introduced for multidimensional evaluation. Results indicated that our model significantly outperformed the general-purpose models in key dimensions such as evidence alignment and comprehensibility, highlighting its potential for patient-clinician communication and personalized patient education. In addition to the automated referee evaluation, a small-scale human expert assessment was conducted to further validate the reliability of the referee model framework. Across both referee LLMs and 3 human experts, PCaPLMM_SFT was consistently ranked as the top-performing model in all lifestyle domains. Although the level of agreement varied across evaluators, the overall ranking trend remained stable, supporting the robustness of the comparative results.

For the base architecture, we selected Baichuan2-7B-Chat, which offers a balance between parameter scale and open accessibility, enabling targeted optimization and potential local deployment [14]. Unlike prior models that relied on authentic physician-patient dialogues for fine-tuning (eg, HuatuoGPT [26]) or expert-curated annotations (eg, Med-PaLM [23]), this study proposes an automated, literature-driven framework for training data generation. Through semantic chunking, structured indexing, and prompt-based reformulation, we constructed a large-scale, low-cost, and high-quality training corpus covering 5 major lifestyle scenarios. This approach not only enhances the controllability of medical knowledge but also addresses the long-standing challenges of data scarcity due to privacy restrictions and limited willingness for data sharing.

At the task level, PCaPLMM_SFT demonstrates an improved capacity for understanding and generating patient-style language. Compared with the document-focused generative models such as RadOnc-GPT [27], PCaPLMM_SFT emphasizes flexibility and empathy in open-ended dialogue, enabling multiturn interactions that incorporated emotional recognition, health guidance, and behavioral recommendations. These features improve communication effectiveness and user engagement in chronic disease management.

Traditional evaluation of medical QA systems has relied heavily on expert ratings, which are costly, subjective, and difficult to scale. With the emergence of the “LLM-as-a-Judge” paradigm [28-30], recent studies have demonstrated that LLM-based referees can provide stable, multidimensional evaluations at a fraction of the cost and time. Consistent with prior work—including validation by Zheng et al [31] in MT-Bench and Chatbot Arena, which showed that structured prompts and rubric-based scoring enabled LLM referees to approximate expert-level consistency—our results further support the feasibility of automated evaluation frameworks. The 2 referee models (Qwen3-Max and DeepSeek-R1) showed moderate to good agreement across rounds, indicating that standardized rubrics and controlled prompting can reduce evaluation variance. However, the lower stability of DeepSeek-R1 compared with Qwen3-Max may reflect model-specific differences in prompt sensitivity, reasoning behavior, and evidence integration; as a reasoning-oriented model, DeepSeek-R1 may be more affected by prompt wording, contextual structure, and the ordering or redundancy of retrieved evidence fragments, whereas Qwen3-Max may have followed the scoring rubric and formatted output more consistently [17,32,33]. These findings extend previous evidence and reinforce the role of automated referees as a practical, scalable tool for benchmarking domain-specific LLMs and guiding iterative model improvement.

In our expert-LLM consistency analysis, ICC values were in the moderate range, indicating that although the overall ranking trends across models were aligned, finer-grained scoring differed. This pattern is consistent with recent literature suggesting that human experts and LLM judges operate with distinct evaluative priors [10,31,34]. Human experts typically emphasize clinical risk, actionability, contextual relevance, and patient-specific feasibility [35]—dimensions inherently subjective and embedded in real-world clinical reasoning. In contrast, LLM referees tend to weight structured coherence, completeness, and evidence alignment more heavily.

Therefore, the observed discrepancy does not reflect disagreement in model performance but rather divergence in evaluation emphasis. These differences underscore that human and automated assessments are complementary rather than interchangeable. In this study, LLM referees were positioned as scalable, standardized, and reproducible tools for rubric-based evaluation rather than as substitutes for human clinical experts. They can efficiently assess evidence alignment, comprehensibility, relevance, empathy, and feasibility under unified criteria [36], but their scoring may still be influenced by prompt design, model-specific preferences, training corpora, and task framing. In contrast, human experts provide experience-based judgment on patient safety, individualized applicability [37,38], risk boundaries, and practical feasibility [39,40]. Integrating both perspectives enables a more comprehensive evaluation framework and mitigates the limitations associated with relying on either method.

This study offers several methodological innovations beyond conventional SFT- or RAG-based biomedical LLM development: (1) Automated literature-to-QA generation pipeline: we proposed a fully automated workflow that transforms high-quality evidence into structured QA pairs, enabling efficient, standardized, and scalable construction of domain-specific training corpora. (2) A generalizable methodological paradigm for chronic disease management: the modular framework—integrating knowledge slicing, structured QA generation, and domain-adaptive SFT—can be readily extended to other chronic disease contexts, providing a reusable blueprint for developing patient education LLMs. (3) Evidence-grounded evaluation framework using a RAG-augmented referee LLM: we established a structured referee LLM–based multidimensional assessment system, enabling standardized evaluation of evidence alignment, safety, clarity, and relevance.

This study has several limitations. First, although the model’s responses integrate evidence-based medical knowledge, their effectiveness and safety have not been validated in real patient interactions; this remains a major limitation for a patient-facing system and an important prerequisite for clinical translation. In addition, the manual QA audit underscores the need for upstream quality control in automated literature-to-QA generation. Although revised and filtered samples were used for SFT, residual corpus noise may remain. Despite the additional interrater agreement analysis, full-corpus manual validation was not performed. Therefore, PCaPLMM_SFT should currently be regarded as a proof-of-concept and a patient education support tool rather than a stand-alone clinically deployable system. Second, despite using multiple referee models and incorporating human expert assessments, bias related to model architecture, training corpora, or corpus-generation procedures may still persist. Although the referee LLM outputs were examined through interround, intermodel, and human-LLM consistency analyses, the referee scores should be interpreted as auxiliary evaluation results rather than definitive expert adjudications. In particular, alternative QA generators were not systematically tested; therefore, potential generator-induced bias from Qwen3-Turbo may remain. Future studies should compare multiple QA generators and incorporate cross-model review or expert-annotated reference sets to further strengthen corpus diversity, evaluation robustness, and generalizability. Third, the human expert validation was based on a limited sample of 50 QA pairs across 5 lifestyle scenarios. Therefore, the expert evaluation should be interpreted as a supplementary consistency check rather than a fully representative assessment of the entire QA corpus or broader patient-facing use scenarios. Finally, as this work primarily focused on methodological development, real-world behavioral outcomes and long-term follow-up effects could not yet be assessed.

Future work should expand real-world interaction datasets and incorporate patient feedback to enhance external validity. Methodologically, applying stronger regularization, early stopping, and multisource corpora may further improve model generalization. In addition, the failure cases in Table 5 indicate that hallucination risks, including fabricated or irrelevant citations, cannot be fully eliminated by RAG and manual review alone; future work could integrate automated fact-checking modules for claim decomposition, citation verification, and evidence support assessment, while retaining expert review in clinically sensitive scenarios. Given the variability observed across referee LLMs and human experts, establishing a multilayer evaluation system that integrates LLM judges, clinicians, and patient users will be essential for improving the stability and safety assessment of model behavior. Finally, extending the proposed literature-to-QA pipeline and structured evaluation paradigm to other diseases represents an important direction for developing scalable, evidence-grounded patient education LLMs. Further validation with broader Chinese-language clinical and patient education resources is also warranted. Moreover, real-world RAG deployment involving multiple retrieved fragments and multiturn dialogue history may require adaptive chunk selection, history summarization, or longer-context models beyond the context window settings used in this study.

As an SFT LLM developed for healthy lifestyle management in patients with prostate cancer, PCaPLMM_SFT supports the feasibility of combining domain-specific knowledge injection with structured QA-based training strategies. The model generated evidence-supported, comprehensible, and empathetic responses across lifestyle-related dialogue scenarios, suggesting its potential as a research prototype and patient education support tool under appropriate clinical oversight. This work also establishes a feasible and reproducible framework for developing disease-specific generative dialogue systems by integrating automated literature-to-QA construction and dual-layer evaluation. The methodology may provide guidance for adapting similar models to other chronic disease contexts. Moving forward, real-world patient interaction studies, iterative dataset refinement, and multistakeholder human-AI coevaluation will be needed before such systems can be considered for broader patient-facing deployment.