To ensure comprehensive coverage of PICC-related issues, a knowledge base for a PICC chatbot was designed using educational materials employed at the Samsung Medical Center. These materials comprised general procedures for PICC insertion, potential symptoms or complications during catheter maintenance, and precautionary measures for maintenance, including dressing changes and heparin flushing. Additionally, daily activities prohibited or allowed to keep PICC safe were incorporated into the materials based on questions frequently asked by patients. Most content was similar to other hospitals’ educational materials and was based on patients’ supportive care needs from an empirical study [5,19-23].

To build structured consultation flows, two oncology-specialized nurses were invited to generate response scenarios using a binary decision (yes or no) tree format. For example, when a patient indicated fever symptoms, the guided dialogue was configured to elicit the current body temperature and assess whether it surpassed the threshold of 38°C. Subsequently, if the temperature exceeded the threshold, the manual directed the patients to seek immediate medical attention. All figures and tables within the materials were converted into structured text to enhance the efficiency with which the chatbot processed and retrieved relevant information.

A multilayered architecture integrating FT, RAG, and a FSM was developed to progressively strengthen protocol adherence in PICC self-management consultations, as illustrated in Figure 1. In model 1, domain-specific responses are generated through FT. In model 2, responses are further grounded in guideline-based documents through RAG. In model 3, an FSM layer is introduced to constrain the dialogue flow according to predefined clinical decision pathways.

User queries are processed by the LLM to generate a JSON-based intent classification and confidence score. When the confidence exceeds a predefined threshold (≥0.8), the query is routed to the corresponding FSM; otherwise, the RAG pipeline is used as a fallback. Through this incremental design, each layer adds a distinct capability—domain knowledge, evidence grounding, and deterministic control—enabling structured and protocol-adherent medical dialogue.

To isolate the contribution of each architectural component, the three models were constructed incrementally such that each subsequent model retained all elements of the previous model and added a single functional layer.

We employed an expert-panel content-validation approach, in which qualified clinical experts judged whether the LLM-generated responses met the standards of expert-led PICC consultation (Figure 2). This approach follows established methodology for clinical content validation, in which panels of three to five highly experienced experts judge whether an instrument or intervention meets domain-relevant clinical standards [27,28]. It is also the modal practice in published LLM-in-health care evaluation, where median panel sizes of 2‐4 evaluators are typical [29].

The evaluation was conducted by a panel of three nurses specializing in infection control and PICC management, recruited via email between August and September 2025, all with at least 10 years of experience in PICC education, counseling, and clinical care for patients with cancer. A query set of 43 PICC-related questions was adopted from a previously published rule-based PICC chatbot whose questions were derived from real patient inquiries [30]. The 43 queries were balanced across six PICC-management-related topics: (1) catheter care (n=10), (2) symptoms (n=4), (3) insertion site abnormalities (n=8), (4) heparin flushing (n=9), (5) daily activities (n=11), and (6) emergency situations (n=1). All identical queries were sequentially entered into the three models (models 1, 2, and 3), generating 129 responses (Section S2 in Multimedia Appendix 1).

The procedure consisted of two sequential stages: a query-level forced-choice preference evaluation, followed by a model-level global rating across eight predefined evaluation dimensions. For each of the 43 queries, each expert independently reviewed the three responses generated by models 1, 2, and 3 and selected the single response judged most appropriate as PICC self-management guidance. To minimize bias, no information regarding the underlying architecture of each model was disclosed to the experts, and all evaluations were conducted in a blinded manner based solely on the response texts. After completing the query-level evaluation for all 43 queries, each expert reviewed the full set of 43 responses generated by a given model and provided a single global rating for that model on each of eight predefined evaluation dimensions: accuracy, clarity, actionability, completeness, efficiency, adaptability, safety, and overall quality. These dimensions were defined based on prior studies of LLM-powered medical chatbots (Textbox 1) [26-31]. Each rating was provided on a 5-point Likert scale (1=highly inappropriate, 5=highly appropriate).

Following the quantitative performance evaluation of the three models, a qualitative focus group interview was conducted as a debriefing session to provide an in-depth interpretation of the evaluation results. The interview was conducted via the Zoom video conferencing platform (Eric Yuan) with the following objectives: (1) to thoroughly understand how each expert interpreted and applied the evaluation criteria, (2) reach a consensus on items where discordance existed among experts, and (3) identify specific areas for improvement, particularly for items that received low scores. The interview lasted for 1 hour and 10 minutes, and the entire session was audio-recorded with the participants’ consent.

Regarding expert preferences (forced-choice), the frequency of expert-selected models was aggregated to determine the distribution of preferred models per query, and the overall preference rate (%) for each model across all 43 queries was calculated. A winning model for each query was defined as one selected by at least two of the three evaluators. Interrater agreement was quantified with Gwet AC1, which provides more stable estimates than Fleiss’ κ when preferences are imbalanced across categories [31]. Differences in model-preference rates were tested with Cochran Q test, with post hoc pairwise comparisons using McNemar test with Bonferroni correction (corrected α=.0167).

Regarding the global ratings, descriptive statistics were used. All eight domains were presented as mean (± SD). For visual comparison of performance differences among the models, radar charts were constructed based on average scores across the eight evaluation dimensions. All analyses were performed using R (version 4.5.2; Posit PBC). Where inferential statistical tests were conducted, 2-sided tests were used, 95% CIs were reported when appropriate, and P values <.05 were considered statistically significant unless adjusted for multiple comparisons using the Bonferroni method.

For the qualitative interview, the recorded material was first transcribed verbatim. Inductive content analysis was subsequently performed on the transcribed data to extract expert rationale and contextual examples that supplemented the quantitative findings. One researcher (ML) organized the coded content, and a second researcher (SB) independently confirmed the coding; discrepancies were resolved by consensus, with adjudication by a senior author (JY) where needed. Through this analysis, we identified the rationale underlying the quality of PICC consultation and the key elements for LLM-based chatbot improvement. The eight evaluation dimensions provided a loose initial framework for organizing codes, but additional themes were allowed to emerge from the narratives.

This study was reviewed and approved by the Institutional Review Board of Samsung Medical Center (Institutional Review Board number SMC 2025-07-146). All procedures were conducted in accordance with the Declaration of Helsinki. Informed consent was obtained from each evaluator by email prior to participation. Evaluator identities were replaced with anonymized codes (N01, N02, and N03) in all records, transcripts, and reported quotations. However, all evaluator-derived data were temporarily linked back to evaluators throughout the qualitative debriefing interviews. Each evaluator received an honorarium of 100,000 KRW (≈US $70) upon completion of the evaluation.

Based on the expert evaluations, model 3, an integrated architecture combining FT, RAG, and FSM, was most frequently selected to provide the most appropriate response across all topics in the PICC query set, followed by model 1 (Figure 3). Model 3 was the most frequently winning model for Catheter care (8/10, 80%), Symptoms (3/4, 75%), Insertion site abnormalities (7/8, 88%), Heparin flushing (6/9, 67%), Daily activities (9/11, 82%), and Emergency (1/1, 100%). However, interrater disagreement was observed in certain categories, particularly for Heparin flushing (1/9, 11%) and Daily activities (2/11, 18%).

For the 43 query-level forced-choice preference evaluations across the three models, interrater agreement was moderate (Gwet AC1=0.581, 95% CI 0.417‐0.745; P<.001). Cochran Q test showed significant differences in evaluator preference frequencies across the three models (Q₂=48.2; P<.001). Post hoc exact McNemar tests with Bonferroni correction indicated that model 3 was preferred significantly more often than model 1 (P<.001) and model 2 (P<.001), whereas model 1 and model 2 did not differ significantly (P=.999; Table S1 in Multimedia Appendix 1). Figure 4 presents a detailed expert preference analysis of all 43 queries. Among these, 23 (53%) achieved a unanimous consensus among the three experts, of which 20 (87%) favored model 3. This pattern indicates that the combined architecture incorporating FSM-based dialogue flow control and RAG-supported knowledge was the most frequently preferred option for the majority of clinically relevant queries.

However, for seven queries, at least one evaluator could not identify a preferred model during the initial evaluation and was subsequently requested to make an additional forced choice. This additional round resolved a determinate winner for six of these queries: Q5 (purchasing dressing supplies), Q6 (changing hospitals for PICC care), Q14 (managing numbness), Q15 (insertion site bleeding), Q20 (securing fixation device), and Q41 (sleeping with catheter). For the remaining query, Q30 (appropriate heparin flushing volume), the three models each received one vote and the outcome remained tied even after the additional forced choice.

Finally, interrater disagreement regarding model preference was observed for three queries: Q30, Q33 (use of the affected arm, physical activity, and daily task limitations), and Q34 (need for dressing replacement due to sweating after exercise). This variability suggests differences in clinical interpretations or context-dependent nuances within certain queries.

Across the eight evaluation criteria, model 3 received the highest mean ratings for most dimensions in this evaluation (Figure 5). Model 3’s highest ratings were for Actionability (mean 4.67, SD 0.58) and Adaptability (mean 4.67, SD 0.58), while its comparatively lower ratings were for Efficiency (mean 3.33, SD 0.58) and Clarity (mean 3.67, SD 0.58; Table S2 in Multimedia Appendix 1). Compared with model 3, model 1 achieved a higher score for Efficiency (mean 4.00, SD 0.00 vs mean 3.33, SD 0.58). Model 1 showed intermediate ratings overall, whereas model 2 received the lowest ratings, particularly for Accuracy (mean 1.67, SD 0.58), Completeness (mean 1.67, SD 0.58), and Overall Quality (mean 1.67, SD 0.58).

Through focus group interviews, experts considered three fundamental components during the quantitative evaluation: (1) in-depth assessment of the patient’s problem and its underlying causes; (2) identification and efficient delivery of information that patients need; and (3) provision of accurate, evidence-based, and tailored responses. These elements were consistently emphasized throughout the discussion as essential criteria for evaluating chatbot performance and determining the appropriateness of consultation quality. The discordance among expert evaluations stems primarily from differing perspectives on the optimal balance between the comprehensiveness and efficiency of information delivery. Nevertheless, based on their rationale for high-quality PICC consultations, evaluators identified several notable strengths in model 3. The experts agreed that the FSM-guided dialogue flow was effective in delivering structured and reliable instructions in clinically careful situations.

However, model 3 exhibited some limitations that tempered its overall effectiveness. The most prominent concern was the excessive response length with unnecessary ancillary information beyond the core message. In addition, model 3’s assessment-oriented approach was appreciated for its depth, yet criticized for lacking actionable conclusions. Thus, to achieve an optimal balance between assessment depth and response efficiency, they recommended that the system secure more expert cases that include scenario-specific, detailed questioning, and decision-making.

This study showed that a hybrid architecture integrating FT, RAG, and FSM was preferred by expert evaluators in the development of an LLM-based chatbot for PICC patient consultations. In blinded expert evaluations conducted by PICC-specialized nurses, model 3 (FT+RAG+FSM) achieved a preference rate of 79%, showing higher expert-rated acceptability over single-component architecture. In terms of global ratings, model 3 descriptively received higher Likert scores than the other two models on expert-rated actionability and adaptability.

The FSM-based dialogue management system in model 3 was implemented through a two-level hierarchical structure in which the major domains of PICC care, such as symptom assessment, daily management, and emergency conditions, were defined as high-level states. User queries were mapped to appropriate states via intent classification, after which low-level scenario flows generated responses that aligned with clinical protocols. This structure follows the design principles demonstrated in prior work, showing that decomposing complex task-oriented dialogue into subgoals and distributing decision-making across different levels improves response robustness and task completion [32]. The established principles of medical consultation delineate a sequential framework comprising (1) problem identification, (2) comprehensive patient assessment, and (3) the provision of structured and tailored information [33,34]. Considering that PICC consultations must adhere to these established principles, FSM-based response generation may support more clinically acceptable consultation by serving as a “conversational guardrail” that ensures structured, protocol-aligned interactions [35]. Previous research has shown that in medical consultation scenarios, general LLMs used without procedural flow control based on clinical protocols may skip essential symptom assessment steps or generate clinically unverified recommendations [36,37]. Similar patterns were observed in this study, as models without FSM-based control often deviated from the core clinical intent or produced unnecessary information, highlighting the need for structured flow management.

Contrary to our hypotheses, model 2 (fine-tuned model + RAG) received lower descriptive Likert ratings than model 1 (fine-tuned model alone) on overall quality and most evaluation dimensions. This may be because the contextual volume of the educational materials used in this study was insufficient. In general, the RAG was invented to address the hallucinations of LLMs; however, its performance can be affected by the quality of documents retrieved by LLMs [38]. Notably, when documents contain insufficient contextual information to disambiguate vague or imprecise user queries, contradictory or distracting material is likely to be incorporated during the retrieval process, thereby diminishing the accuracy of LLM-generated responses [39]. Previous research has demonstrated that the RAG system performance declined below that of the baseline model operating without a retrieved context when a single irrelevant document was incorporated into the retrieval set [40]. Although numerous hospitals provide patients and caregivers with PICC education materials, the instructions are typically brief to ensure ease of understanding [19-22]. Hence, it may have been relatively difficult for the RAG, which uses simplified documents, to provide quality and sufficient information based on the retrieved context (poor chunking quality). In addition, our retrieval was configured with top-k=3 without a similarity threshold or no-retrieval fallback. Hence, under sparse and brief source documents, this likely surfaced topically adjacent but non-gold chunks that were nonetheless injected into the prompt (poor retrieval relevance). The combination of these two mechanisms—poor chunking quality and poor retrieval relevance—may explain why model 2 fell below the model 1 baseline rather than defaulting to it. From this perspective, our findings suggest that integrating FSM frameworks into LLMs may mitigate existing technological limitations by establishing more deterministic conversational pathways tailored to individual patient inquiries, even when the underlying document repository is insufficient.

Despite its strengths, model 3 exhibited difficulties with several query types. These queries tend to involve context-dependent considerations tightly linked to real-world daily routines or individualized patient circumstances, domains in which flexible reasoning beyond the constraints of FSM rules is often required. Moreover, these cases frequently lack a single correct answer within the clinical guidelines, requiring experiential judgment from experts. Variability in expert interpretation, shaped by differing clinical priorities, such as emphasizing reassurance versus patient convenience, also contributed to disagreement in those cases. A related concern raised during expert evaluation was conversational efficiency. This is because the current FSM design elicits one piece of information per turn; clinically straightforward scenarios sometimes required more dialogue steps than necessary. A potential solution is multislot extraction, in which the LLM is prompted to identify multiple relevant entities (eg, symptom type and severity) within a single conversational turn, thereby reducing the number of required exchanges while preserving protocol adherence. This optimization, combined with adaptive turn-skipping logic that bypasses redundant assessment steps when sufficient information is already available, represents a promising direction for improving the practical usability of FSM-guided dialogue systems.

From a technical standpoint, the development of FSM-based states and transition rules requires substantial expertise and iterative refinement. Mapping diverse query types to predefined states and specifying appropriate transition conditions necessitate close collaboration between clinical experts and system developers, which may limit scalability and increase the system maintenance burden. To address these challenges, future systems may benefit from three key strategies: (1) proactive collection of expert-derived real-world cases, (2) transformation of these cases into formalized ontologies, and (3) the development of interpretable reasoning modules grounded in such ontologies.

Ontologies can structure hierarchical and semantic relationships between clinical concepts, partially automating intent classification and state mapping, while enhancing system extensibility [41-43]. When integrated with standardized medical ontologies, the reliance on manually defined rules can be reduced while preserving clinical accuracy. Furthermore, ontology-based approaches offer reusable knowledge structures that support efficient scalability beyond PICC management to other clinical domains.

This study had several limitations. First, the evaluation was based on human expert assessments, and no benchmark dataset currently exists for validating PICC-related chatbot performance. The evaluation criteria were adapted from prior studies but were not derived from a formally validated instrument, limiting the ability to derive fully objective quantitative measures. Accordingly, future research should prioritize the development of dedicated evaluation metrics for LLM-mediated clinical consultations or, alternatively, adapt established frameworks, such as the Calgary–Cambridge Guide, to assess adherence to the principles of effective patient–physician communication [33,34].

Second, although the assessment was conducted blinded, true blinding may have been severely compromised by the structurally distinct outputs of the FSM-guided model, which generated noticeably longer responses with multiturn back-and-forth interactions. While the evaluators received no prior briefing on the underlying technical mechanisms, the structural distinctness of the FSM-guided outputs may have inadvertently signaled the architecture in some cases. The expert preference for the FSM-guided model should therefore be interpreted with this caveat.

Third, model performance was constrained by the scope of the preconstructed educational materials used for training. Although these materials sufficiently cover core PICC management tasks, the system may be limited in responding to diverse real-world scenarios encountered in everyday life. Therefore, additional strategies are required to support flexible reasoning in unexpected and context-rich situations. In addition, the knowledge base and FSM transition rules were derived from educational materials and clinical workflows of a single tertiary cancer center. Institution-specific elements such as dressing intervals, role permissions for heparin flushing, and local escalation pathways may not generalize to settings with different staffing models or referral practices. The 43 evaluation queries, although derived from real patient inquiries, were collected at the same single cancer center, which may also limit their generalizability. Multicenter validation would be required before the proposed architecture can be considered clinically generalizable.

Fourth, the RAG chunking strategy used in this study relied on double-newline (paragraph-level) delimiters matched to our institutional materials, which were preformatted with consistent paragraph breaks. Given that PDF text extraction can introduce irregular or broken newline characters, this approach is sensitive to document formatting and may have reduced chunking quality. A more sophisticated, layout-aware parsing strategy might therefore have yielded different baseline results for model 2, and the chunking approach would need to be adapted before applying this architecture to external document sources.

Fifth, the evaluation relied solely on health care professionals and may not have fully reflected patient-perceived usefulness or usability. Establishing content validity by qualified clinical experts was a deliberate precondition of this study. It is crucial to ensure the safety of content before it is delivered to patients and to enable subsequent patient-experience data to be interpreted against a clinically validated baseline. Building on this expert-validated baseline, future work should extend the evaluation to include patient-facing usability assessment together with operational performance metrics such as system latency, token costs, and fallback frequency.

In conclusion, an FSM-guided RAG architecture (model 3) was preferred by oncology nurses in 79% of PICC self-management scenarios over a fine-tuned–only model (model 1) and a fine-tuned+RAG model (model 2). Model 3 received descriptively higher expert-rated acceptability scores on actionability, adaptability, and safety, while showing lower ratings on efficiency, possibly reflecting its multiturn structure. Model 2 received descriptively lower Likert ratings than model 1 on most dimensions, although the two single-component architectures did not differ significantly in 43-query model-preference rate after Bonferroni correction. These results indicate that retrieval augmentation alone, without an additional structural constraint, did not yield a robust improvement in expert-rated acceptability in this single-center evaluation. These findings support the incremental value of FSM-based dialogue control as a complement to FT and a RAG system. By constraining the conversation flow to state transitions based on clinical evidence, the chatbot’s output becomes more closely aligned with established expert PICC consultation patterns. While this study did not assess patient-facing usability, our findings suggest the potential to provide more clinically plausible virtual consultations to patients.