Gastric cancer is the most common gastrointestinal tumor worldwide, ranking as the fifth most prevalent cancer and the fifth leading cause of cancer death globally [1]. Decreases in 1 or more domains (physical, role, social, etc) of functioning and an increase in negatively associated symptoms (dysphagia, reflux, etc) were found perioperatively among the patients with gastric cancer [2]. Furthermore, if patients are unfamiliar with managing these symptoms or lack adequate disease information, these problems may worsen. In addition to the impact of symptoms on the human body, related economic costs and psychological burdens also affect patients’ recovery [3]. Therefore, it is crucial to provide patients with gastric cancer with adequate health education information for effective overall management.

Patients and their families frequently encounter difficulties in accessing reliable and comprehensible disease-related information [4]. This gap is primarily due to the limitations of traditional health education methods, such as pamphlets, verbal instructions, and infrequent consultations, which often fail to meet the dynamic and personalized needs of patients with gastric cancer. Besides, these kinds of health education methods often include too much content with dense text, which may lead to information overload. Patients will feel confused when first facing a large amount of information and even more anxious and fearful with diagnosis and treatment. Patients with gastric cancer also face unique challenges due to the complexity of the surgery, subsequent symptoms, and perioperative care. Current health education resources and methods are insufficient to meet the demand for personalized, real-time responses, which proves to be a catalyst for innovative solutions.

With the development of technology, some patients seek online medical information when they feel uncertain or concerned. Nevertheless, such behavior sometimes may entail potential risks such as exacerbating anxiety and inappropriate self-management due to unverified information [5]. Digital health interventions, particularly conversational agents—such as chatbots or virtual assistants, offer promising solutions. Research has proved that conversational agents had the potential to significantly improve patient care by enhancing symptom management, supporting self-management, and providing patient support [6-8]. Gomaa et al [9] developed an SMS text messaging–integrated and chatbot-interfaced self-management program for patients with gastrointestinal cancer undergoing chemotherapy. The authors noted that the omission of natural language processing (NLP) capabilities restricted deeper interactive engagement, and future research should explore integrating NLP to enhance responsiveness to diverse patient expressions. Naseri et al [10] examined the effectiveness of 2 AI chatbots, Sider Fusion AI Bot and Perplexity AI, in improving patient outcomes, alleviating anxiety, and promoting informed decision-making, highlighting the importance of tailored communication styles to enhance patient engagement and outcomes. Thus, an AI-based conversational agent is suggested to use in our study.

However, generic and unverified AI-based conversational agents may pose potential health risks such as delivering inaccurate information that seems convincing [11]. Their integration into clinical practice is sometimes hindered by inherent limitations, and most refers to the phenomenon of “hallucinations” and unclear boundaries of responsibility and ethics [12]. A growing number of AI-driven platforms have been developed for high-prevalence malignancies, such as breast, prostate, and lung cancers, but rigorously designed conversational agents specifically tailored for gastric cancer remain scarce [13]. In the high-stakes context of gastric cancer care, where erroneous dietary or postoperative advice can lead to severe clinical complications, ensuring information fidelity is paramount. RAG has emerged as a robust architectural solution to mitigate these risks [14]. Zhou et al [15] developed a Chinese gastrointestinal disease chatbot and demonstrated the innovative potential of Retrieval-Augmented Generation (RAG) technology. The retrievable knowledge functions as a form of nonparametric memory that is easily updatable, capable of incorporating extensive long-tail knowledge, and suitable for encoding confidential information. Consequently, as the landscape of AI-based conversational agents continues to expand, the focus of development must transcend mere conversational fluency. Clinical safety must remain the cornerstone of AI deployment to ensure that these intelligent agents serve as a secure bridge between complex medical knowledge and the patient’s daily recovery.

Based on the clinical background and gap, 4 research questions are discussed in this study as follows: (1) How to explore patients’ needs and suggestions on the current gastric health education mode? (2) How to use large language models (LLMs) and RAG to develop a conversational agent for patients with gastric cancer? (3) How to evaluate the performance of the agent in terms of accuracy and usability? (4) How to reflect the user experience among patients and other related stakeholders regarding the conversational agent as a tool for patient education?

The aim of this study was to develop and implement an AI-guided conversational agent for patients with gastric cancer. We adopted an action research approach to gain insights from patients and stakeholders, develop the conversational agent, evaluate its usability and accuracy, and gather user feedback by using qualitative and quantitative methods.

An action research design was adopted, which is a collaborative, democratic approach and process where research participants are collaborators rather than participants [16,17]. Unlike conventional approaches, action research emphasizes collaborative knowledge building and social change to develop contextually relevant interventions [18]. It has been applied by health care professionals to improve both patient experience and working conditions of those who deliver care [19,20]. The implementation of digital health interventions presents unique challenges in contrast to other studies [21]. To gain the most from digital health products such as conversational agents in our study, we should clearly know what is really needed in practice [20]. Co-design or collaboration with all stakeholders provides effective means for developing and implementing digital health interventions that suit the needs of the end users [22].

Action research emphasized that each study should adapt the action research framework according to its specific research aim and content. Our study followed an iterative process adapted from Lewin’s [23] 4-step cycle of action research (Figure 1). The reporting guideline and checklist (Checklist 1) have been used to report our study [24].

The multidisciplinary team was composed of 3 advanced nurse specialists, 2 surgeons, 2 researchers, and an information technology engineer. Each team member played a distinct and essential role in the development and implementation of the health education conversational agent. The advanced nurse specialists and surgeons who had 7-20 years of experience in the gastrointestinal field brought rich clinical experience in gastric perioperative care and patient education. They were responsible for identifying key health education weaknesses and constructing a knowledge base for the conversational agent. The researchers led the overall research design, data collection, and analysis process. The information technology engineer was responsible for conversational agent development, program coding, and other informatics work. The action team functioned as a partnership, characterized by shared decision-making and collaborative effort throughout all phases of the research.

This study was conducted at the Gastric Cancer Center of Zhongshan Hospital in Shanghai, China, a facility known for its advanced medical care devices and patient-centered care delivery. Patients with gastric cancer and relevant stakeholders were involved in this action research study.

Four phases were designed to realize the triangulation of research by involving multiple participants and combining qualitative and quantitative approaches [25].

Data collection was conducted from September 2023 through April 2025 across 4 sequential phases. Participants were recruited using convenience sampling from the gastric cancer center of Zhongshan Hospital. Eligible participants were identified by action research team members including patients, clinical nurses, nursing managers, surgeons, clinical psychologists, and dietitians (recruitment criteria in each phase are shown in Table S4 in Multimedia Appendix 1).

Qualitative interviews were recorded using a digital voice recorder with participants’ permission. Audio recordings were transcribed verbatim into text immediately after each interview. Data collection ceased until no new themes or meaningful information emerged from subsequent interviews. Inpatients who used the conversational agent were invited to finish the validated CUQ on discharge day. CUQ is a chatbot-specific usability questionnaire that is comparable with the Systems Usability Scale [27]. It consists of 16 balanced questions related to different aspects of chatbot usability. Odd-numbered questions relate to positive aspects of the chatbot, and even-numbered questions relate to negative aspects. All 16 questions are scored using a 5-point Likert-type scale. Each item was scored on a 5-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree). Scores are calculated out of 100. Besides the CUQ, the conversational agent was also tested by 1 more question about the content relevance. The additional question “Do you think the responses provided by the conversational agent appropriately match your question?” was also rated on a 5-point Likert scale as CUQ, and this score was analyzed separately. Face validity was assessed among 2 patients and 1 nurse.

The accuracy and RAG knowledge hit rate are 2 critical metrics for analyzing the performance of question-answering systems [28]. The accuracy rate was evaluated by 5 team experts (3 advanced nurse specialists and 2 surgeons) independently; each response was rated as correct or incorrect, and it was considered accurate if at least 4 of the 5 experts judged it to be correct. Overall interrater agreement across all 3 rounds of evaluation was calculated to verify the consistency of the expert assessments. The RAG knowledge hit rate determines whether the system can accurately retrieve relevant medical information from the knowledge base, preventing answer errors caused by missing knowledge or retrieval failures while mitigating the risk of large-model “hallucinations.” It was calculated directly by the system. We combined both automated retrieval metrics and manual assessment to offer practical and adequate evaluation results.

The use of AI-based conversational agents in health care raises important ethical issues, particularly regarding data privacy, transparency, and patient autonomy. The conversational agent was designed to support, rather than replace, professional medical advice, in order to minimize the risk of overreliance on AI-generated information. The system repeatedly emphasizes that qualified health care professionals’ advice should be regarded as the final authority. Absolute or definitive language is deliberately avoided in responses, especially in certain sensitive topics such as diagnosis and treatment. Besides, conversations were anonymized and stored in encrypted files accessible only to the research team.

This study was approved by the Medical Ethical Review Board of Zhongshan Hospital Fudan University (B2022-613R). Individual informed consent was obtained from all participants, whether in the qualitative interviews or the quantitative researches. Participants were also informed that their personal identifiers were removed from the research database and they were voluntary to withdraw from the project at any time without giving a reason. This research complies with the Declaration of Helsinki and conforms to the data protection guidelines outlined by the local governing bodies.

We recruited 15 patients (T1-T15) in the last phase who finished the CUQ and 8 nurses to do the in-depth interview to deeply explore the factors facilitating and hindering the conversational agent. Detailed characteristics of 8 clinical nurses are shown in Table 5.

Emerging themes from the qualitative study results were mapped on the 3 domains of the technology acceptance model (Table 6). Detailed information about the qualitative analysis is shown in Text 2 in Multimedia Appendix 1.

This study demonstrates how the action research framework can be a practical methodology to develop a health education conversational agent, comprising the diagnose and plan, act and implement, evaluate, and reflect phases. These 4 phases formed a continuous and iterative process in which the findings from each phase informed the subsequent stage. This user-centered approach enabled the systematic integration of patient needs, clinical expertise, technical implementation, and iterative evaluation, and offered an approach for developing digital health interventions that are both clinically relevant and acceptable to end users. By placing the diagnose and plan phase at the beginning of the process, researchers were able to identify real clinical requirements rather than technological assumptions and avoid certain barriers to adoption at the earliest stage. The act and implement phase realized outcomes that the conversational agent not only incorporated advanced technologies but also remained clinically practical and aligned with routine health care delivery by involving multidisciplinary stakeholders. The successful implementation of technologies such as RAG also relies on close collaboration between medical and informatics specialists. The evaluate and reflect phases constituted essential components of the iterative optimization process. Integration of quantitative and qualitative findings enabled developers to identify specific directions for system refinement and provided a foundation for subsequent iterative development. Overall, the action research–based development pathway in our study may offer opportunities for improving the sustained adoption of digital health interventions in real-world clinical settings through iterative, user-centered, multidisciplinary, and mixed methods principles.

Health education is a cornerstone of therapy and recovery for patients with physical and mental conditions [35]. Conversational agents for health education are necessary because there may be a discrepancy between patients’ actual knowledge needs and what clinical practitioners consider necessary [36-38]. For patients, this digital human serves as a convenient and timely resource tool, providing accurate and personalized information, particularly when clinical staff are unavailable. Lancet Digital Health once mentioned the urgent concern regarding whether existing tools and digital health interventions are appropriate for the aging population [39]. In our study, we emphasized this problem and found that the simplicity and convenience of this chatbot bridged this gap based on the results of phase 3 and phase 4. In usability questionnaires and in-depth interviews, almost no patients mentioned that the system was difficult to use.

The design of the nurse avatar and nonverbal behavior received many praises which proved that embodied representation did have significant impacts on patient satisfaction and health outcomes [40]. This was explained in the research by Bickmore et al [40] that human relationships are fundamentally formed in face-to-face conversation and nonverbal behaviors are vital for facilitating social connection and understanding. However, the input and output modalities may require further optimization. Contrary to our expectations, some end users experienced confusion and discomfort when using the voice input to initiate interactions. Patients with limited verbal expression skills or those speaking regional dialects faced difficulties in recognizing their questions accurately. In addition, the chatbot currently provides responses primarily in text format. Incorporating multimedia content, such as images and videos, may further enhance users’ comprehension and reinforce health education.

Health education content should be dynamically adapted to individual patients’ health literacy levels to maximize relevance and effectiveness. Personalization of our conversational agent remains limited. Some conversational agents integrate patients’ appointment schedules and personal health information to deliver more personalized health services. However, the incorporation of private data inevitably amplifies concerns regarding privacy and data protection. Fournier-Tombs and McHardy [41] listed a list of potential ethical risks in conversational chatbots, including discrimination, stereotyping, exclusion, lack of privacy, poor data governance, stigma, error tolerance, and overconfidence. Specifically, our team set safety guardrails to remind patients that clinicians’ judgments should be regarded as the final suggestion in medical decision-making.

In our study, stakeholders were also involved among the 4 phases to provide diverse perspectives on the implementation of our conversational agent in clinical practice. Through quantitative and qualitative research, we found that for medical professionals, especially frontline staff, integrating a domain-specific conversational agent into the gastric cancer clinical workflow amidst growing clinical workloads can streamline patient-clinician communication. By alleviating such burdens, clinicians were allowed to concentrate on more complex tasks, such as observing patient conditions. Moreover, clinical experts also expressed interest in leveraging conversational agents for complex tasks such as data analysis. These findings represent important directions for future development in clinical practice.

Advancing health equity worldwide is increasingly drawing attention in recent years [42]. Although medical chatbots are gaining popularity, there is still a dearth of chatbots specific to gastric cancer health education in resource-constrained regions [43]. Researchers and governments have a responsibility to design and implement fair, unbiased, and easily accessible tools that prioritize the needs of patients [44], and our conversational agent provided a promising solution. Once the health education conversational agent in our study can be put into formal use after effectiveness validation, shared professional knowledge bases will be available for regions’ lack of medical resources, thereby improving the health literacy of local patients, promoting health equity, and eliminating health disparities [45].

Enthusiasm for conversational agents as means for health education was revealed among patients and clinical staff in the first phase of our study, which was also proved in other research [46]. Multiple research studies used theory frameworks to develop such digital interventions. For example, a design science research methodology was used to develop a conversational agent for enhanced self-management after cardiothoracic surgery and verified its acceptability [47]. Similarly, our study adopted an action research approach by realizing the user-centered design and usability principles.

Research gaps remain in gastric cancer–specific chatbots although numerous mature conversational agents have been developed in the health care area. Gomaa et al [9] developed a self-management program in patients with gastrointestinal cancer undergoing chemotherapy and used a mixed methods design to evaluate its effectiveness. However, this platform relied on a structured approach instead of NLP, which limited its understanding of nuanced patient contexts. Zhou et al [15] developed a gastrointestinal disease chatbot using RAG and evaluated its performance with the Retrieval Augmented Generation Assessment framework, which provided an important technical foundation for our study. In contrast to the study by Zhou et al, we enriched our research perspectives by adding qualitative results. Similarly, Kim and Park [48] developed a chatbot for patients with gastric cancer undergoing radical gastrectomy and evaluated both user experience and system performance. The accuracy of our conversational agent is comparable with this similar chatbot. An embodied conversational agent may potentially improve engagement by providing additional motivational and emotional support [49], while also providing a foundation for multimodal interaction [50]. Such design elements remain uncommon in existing health-related conversational agents, especially those targeting gastric cancer care. Besides, an evidence-based corpus, RAG technology, model fine-tuning, and safety guardrail mechanisms were implemented in our study to reduce the risk of generating misleading or unsafe information and improve the reliability of the chatbot’s outputs.

Although the study’s strengths lie in its novel combination of digital health intervention development and action research approach, leveraging both quantitative and qualitative data, some limitations must be acknowledged. This study reported only a single cycle of the action research framework. Additional cycles are necessary to iteratively refine the system and evaluate its effectiveness in real-world settings. Although prior studies suggested that subjective ratings and expert evaluations can provide valuable insights into revealing potential risks in real-world use [51], accuracy rate, RAG knowledge hit rate, and usability score were used to test the performance of the chatbot, which was not enough. The complexity of health care contexts and variability in expert judgment may produce bias and inconsistency [52]. Besides, this study was conducted with a relatively small sample from a specific center and cultural context. Stakeholder perspectives were collected during a limited period, which may not fully capture evolving needs and long-term experience over time. In this study, we focused more on the development and implementation of the conversational agent rather than its clinical effectiveness; the impact of the intervention on patient outcomes and other medical facilities remains unknown without multicenter randomized controlled trials.

Last but not least, algorithmic limitations need further attention. The conversational agent was developed using OpenMEDLab2.0 [31], whose technical description is currently available primarily as a preprint rather than a peer-reviewed publication. Although the model was selected because of its medical domain-specific design, future studies should further evaluate and compare its performance with other peer-reviewed medical foundation models across diverse clinical tasks and settings. Furthermore, the current workflow requires enhancement to support functional extensions, such as follow-up questionnaire delivery. Moreover, the system primarily retrieves answers from a predefined knowledge base without incorporating expert clinical reasoning, highlighting the need for further development of chain-of-thought capabilities.

This study provided insights into how the action research approach can inform the development and usability assessment of a gastric cancer health education conversational agent. It also illustrated details of how technologies such as RAG, model fine-tuning, and safety guardrail technologies can be integrated into the conversational agent. Promising results regarding accuracy and usability were demonstrated through both qualitative and quantitative research. However, additional assessments and improvements are warranted to confirm the effectiveness, safety, and long-term adoption. Enhancements such as personalization, multimodal data integration, and advanced reasoning capabilities should also be explored in further studies.