We conducted a single-arm feasibility and acceptability study using an embedded mixed methods design (QUAL [quan]). In this design, the qualitative strand served as the primary method of evaluation, while quantitative survey data and technical metrics provided supplementary context for the qualitative findings (Figure 1). The conduct and reporting of this study were guided by the Good Reporting of a Mixed Methods Study (GRAMMS) checklist [20]. The study was guided by 2 established frameworks: (1) the World Health Organization’s monitoring and evaluating digital health interventions (M&E DHI) framework for assessing usability and feasibility [21] and (2) the theoretical framework of acceptability (TFA) to evaluate constructs such as affective attitude, burden, coherence, perceived effectiveness, self-efficacy, and ethicality [22] (Table 1).

Participants were recruited via purposive sampling from the CGH HMU tele-support program. Patient inclusion criteria required individuals to be 21 years of age or older, have a clinical diagnosis of T2DM, be smartphone users, and demonstrate proficiency in English. Telecarers were eligible if they were registered nurses with a minimum of 6 months of experience in the HMU tele-support program and used the chatbot-generated summary reports during the study period.

The study was conducted from November 2023 to October 2024. Qualitative data were collected through semistructured interviews lasting approximately 45 to 60 minutes. Interviews were conducted in English, either in person or via Zoom, by researchers (RHSO and JWN) who were not involved in the participants’ direct clinical care or telecarers’ work to minimize social desirability bias. The interview guides were mapped to the TFA and M&E DHI domains. All sessions were audio-recorded and transcribed verbatim for analysis. Quantitative data were collected via online closed-ended surveys hosted on Qualtrics. The survey component is described according to the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) [23]. A survey link was sent to enrolled participants, together with a unique participant code to identify unique responses. Participants were informed that the survey would take approximately 15 to 20 minutes to complete. The survey comprised 11 pages, with each page containing between 1 and 8 items. Survey completeness was ensured by using forced responses on all items. A back button was enabled to allow participants to review and change their answers. Survey items were self-developed by the research team and mapped to TFA and M&E DHI framework domains (Multimedia Appendix 1). Patients additionally self-reported their digital literacy status (assessed as very good/good/moderate/poor) and prior experience with using chatbot applications. Technical usage logs, including completion rates and interaction timestamps, were extracted from the chatbot database.

The SingHealth Centralized Institutional Review Board approved the study (reference 2022/2374). Written informed consent was obtained from all participants, and data were deidentified for analysis. All costs are reported in Singapore dollars (SGD). Participants who completed all study activities were given a token of appreciation (SGD 80; the exchange rate at the time of the study was approximately 1.00 SGD=US $0.74).

The chatbot was co-designed with clinicians, telecarers, chatbot developers, system engineers, content designers, and researchers. Its conversation flow was derived from a retrospective review of 852 HMU telecare transcripts (17 telecarers and 173 patients), from which common questions, responses, and educational topics were extracted to build a clinically guided protocol aligned with existing HMU workflows and guidelines. The branching logic defining the follow-up questions, relevant advice, and educational content was co-designed with telecarers to ensure a coherent progression of questions based on each patient’s prior responses. Summary report templates were likewise co-designed with telecarers so that prioritized issues, follow-up flags, and key patient inputs could be easily reviewed without adding unnecessary preparation time before calls.

The chatbot was developed as a message-based interactive agent system using Telegram Messenger (Android version 9.6.0; Apple iOS version 8.9), allowing patients to receive prompts, multimedia guidance, and reminders. The dialogue engine was built on the Botpress framework (version 12.30.9), which handled message parsing, semantic intent extraction, the recording of customized dialogue loops, and the maintenance of session continuity. All dialogue logic was predefined to ensure that responses adhered strictly to CGH tele-support protocols and clinical guidelines. An SQL-based dialogue and scheduling database within Botpress stored conversation data and patient metadata, while the dialogue scheduler coordinated communication between patients and telecarers, maintained timelines for the planned sessions, and triggered automated reminders. The system incorporated active engagement functions such as interactive answer-selection buttons with predefined response options and dynamic branching based on patient responses. The chatbot and its summary reports were hosted on Amazon Web Services (AWS) (Ubuntu version 22.04) with secure access (unique ID and password) and data handling within the AWS environment. All platforms used (Telegram, Botpress, and AWS) are commercially or publicly available, allowing other institutions to reproduce a similar architecture, subject to their own compliance and data governance requirements.

Iterative prototyping and internal testing cycles with telecarers and an individual with T2DM in HMU led to refinements in usability, content accuracy, and report formatting prior to this pilot evaluation. These predeployment cycles were not a formal study, and the present study represents the chatbot’s first formal evaluation in a real-world tele-support context with patients. The chatbot workflow and question group distribution across the 7 sessions are detailed in Figure 2 and Multimedia Appendix 2, categorized by the tele-support schedule: (1) question group 1 preceded call 1, (2) question groups 2 to 4 preceded call 2, and (3) question groups 5 to 7 preceded call 3.

As a feasibility study, no formal sample size calculation was conducted. A target of 20 participants was set, aiming for at least 9 interviews across both patients and telecarers, consistent with empirical guidance on sample sizes for qualitative research [21,24,25] and with early-stage digital health feasibility pilots, where small qualitative samples are typical [11,26,27].

Qualitative transcripts were analyzed using thematic analysis in Dedoose [28]. Three coders (RHSO, LZAT, and MHWG) familiarized themselves with the 14 transcripts through repeated reading. To develop the codebook, 3 transcripts were coded by all 3 coders, prioritizing interpretive depth and analytical judgment over interrater reliability metrics. Discrepancies were discussed and reconciled through iterative team meetings, during which disputed codes were reviewed against the raw transcript excerpts and refined to best represent the data. This process informed the inductive formation of a preliminary codebook, which was also guided by deductive elements drawn from the TFA and M&E DHI constructs to maintain alignment with the study aims. The remaining transcripts were then divided among the 3 coders for independent coding, and themes and codes were further refined through iterative team discussions until final consensus was achieved. Mixed methods integration was conducted with the 14 participants (7 patients and 7 telecarers) who completed both phases, ensuring that integration drew on linked qualitative and quantitative data from the same individuals. The quantitative strand served a supplemental, descriptive role consistent with the QUAL (quan) design [29-31], while the qualitative transcripts were the primary focus of analysis [32]. We integrated the 2 datasets using a narrative approach and a joint display table [33], through which quantitative data supplemented and contextualized the qualitative themes across evaluation domains.

Of the 19 participants enrolled (11 patients and 8 telecarers), 5 were noncompleters who were uncontactable or unable to participate due to time constraints. Therefore, 14 participants (7 patients and 7 telecarers), who completed both the quantitative surveys and the interviews, formed the analytic sample for this study (Table 2). The patient cohort (n=7) had a mean age of 45.3 (SD 15.4) years and reported a median diabetes duration of 10 years, with an IQR of 3 to 25 years and a range of 1 to 30 years. All patients were identified as digitally literate and smartphone users, and 2 participants noted prior experience with chatbots. The telecarer group (n=7) consisted of female registered nurses with a median of 5 (IQR 4-9) years of experience in telehealth (range of 4‐12 y).

The technical metrics for 7 patients showed a median session completion time of 2 minutes 58 seconds (IQR 2 min 12 s to 5 min 27 s). Patients completed 79% of the sessions, and 57% (n=4) completed all 7 sessions. Technical disruptions were minimal, with 3 time-outs recorded across the study period (1 in session 1, 1 in session 2, and 1 in session 5). Chatbot usage patterns indicated that 57% (n=4) of patients accessed at least one educational content item, and 86% (n=6) used the chatbot to flag specific topics for their upcoming tele-support calls.

All enrolled participants completed the quantitative survey, with no duplicate responses received. The surveys were also fully completed, with no incomplete items. Overall satisfaction was high among patients, with 86% (n=6) reporting they were satisfied and would recommend the chatbot. Telecarers’ findings were more mixed, with 43% (n=3) supporting a broader rollout of the system to patients in other HMU programs or outpatient clinics. The participants’ full survey responses are available in Multimedia Appendix 3.

In the following sections, we present an integrated analysis of qualitative and quantitative findings across each evaluation domain (Table 3; Multimedia Appendix 4).

This pilot evaluated the feasibility, usability, and acceptability of “Siu Dai,” a clinical protocol–guided chatbot designed to support T2DM self-management within a hospital tele-support program. While findings noted high patient acceptability and functional support for tele-support delivery, integration into clinical workflows showed mixed results.

Patients reported high technical usability, describing the interface as intuitive and convenient. However, the perception of the brief chatbot sessions as repetitive and lengthy may reflect the influence of content relevance and cognitive load on subjective time perception, whereby routine or low-novelty content leads to self-reported overestimation of duration rather than reflecting the objective length of the sessions [34,35]. Similarly, while telecarers found the automated chatbot summaries accessible, they identified a need for greater contextual granularity (eg, patients’ living and caregiving arrangements) to support clinical interpretation. These results indicate that while technical usability was present, future iterations must prioritize content and interaction design optimization to bridge the gap between usability and clinical utility. This tension reflects divergent priorities between patients, who favor brevity, and telecarers, who require depth for clinical decision-making. This conflict may not be fully resolvable through interface design alone, but instead requires deliberate trade-off decisions in future iterations.

The availability of precall responses enhanced the tele-support services by facilitating more focused dialogue and better prioritization of patients’ needs, but these gains were counterbalanced by increased administrative burdens, specifically telecarers’ concerns about having to navigate separate reports. This might have contributed to the limited endorsement for universal adoption among telecarers, echoing broader literature that highlights poor integration into existing health care systems as a key barrier to the adoption of digital health tools [36]. While initial co-design sessions with telecarers informed development, implementation uncertainty suggests that co-design must be a longitudinal, iterative process rather than a static predeployment phase, though achieving this in practice remains challenging due to resource, time, and technical constraints that are common in digital health implementation [37]. Future integration with electronic medical records (EMRs) might present an opportunity to resolve these interoperability hurdles and streamline data access, though such technical scaling was beyond the scope of this study.

Patient acceptability was high, partly anchored in the institutional credibility of CGH, a finding echoing the literature on systemic trust as a determinant for acceptance of health chatbots [38,39]. Despite this trust, participants highlighted the “human touch” as an irreplaceable component of care. Relational elements, such as empathy, emotional support, and the navigation of complex clinical nuances, were viewed by patients as beyond the current capabilities of the Siu Dai chatbot. This aligns with a growing consensus among users that digital tools are perceived as valuable adjuncts rather than clinical substitutes [36,40,41]. Our findings support this complementary care model and are consistent with the broader consensus in the digital health literature on the clinical use of chatbots, large language models, and AI-assisted care delivery [19,42], whereby routine tasks are automated to allow health care professionals to focus on complex clinical management and relationship-building. Importantly, this study provides empirical support for this principle within a hospital-based tele-support context.

Perceptions of efficacy were influenced by participants’ T2DM literacy levels and experiential knowledge. While newly diagnosed patients found the content helpful and instructive, those with extensive self-management experience noted content redundancy, a common critique among high-literacy users of digital health tools [43]. Telecarers expressed a similar professional ambivalence. They acknowledged the chatbot’s role in facilitating personalized dialogue but remained cautious regarding its impact on longitudinal clinical outcomes and care quality. Despite these reservations, both patients and telecarers recognized the chatbot’s efficacy in enhancing T2DM care-related dialogue and education. Additionally, the chatbot demonstrated “proof-of-concept” for behavioral nudging, as evidenced by a patient’s reported dietary modification following the chatbot’s advice, showcasing the promise of digital coaching in supporting behavior change in T2DM management [44]. Still, most participants viewed the chatbot as supplementary rather than transformative due to its one-size-fits-all approach. While clinical protocol–guided chatbots ensure standardized, evidence-based care delivery, they lack the algorithmic tailoring required to address personalized or complex care needs [36,45]. To move from a supplementary to a transformative role, future iterations must resolve this gap between standardized clinical protocols and users’ needs, ensuring that content remains relevant throughout the patient’s care journey.

A primary strength of this study is the application of established digital health frameworks, the TFA and the M&E DHI, which contributed to the rigor of the evaluation. Additionally, the inclusion of both patients and telecarers provides a dyadic perspective that is often missing in early-stage chatbot research [10,36]. Third, the embedded mixed methods design (QUAL [quan]) allowed quantitative data to supplement and contextualize the qualitative findings. This integration addressed calls in the literature for more mixed methods research study designs to better understand the complexities of digital health in T2DM care [10]. However, limitations should also be noted. As a feasibility pilot study, clinical effectiveness was not assessed, and larger controlled trials are necessary to determine the impact on long-term T2DM outcomes. An additional limitation is that the patient sample was highly digitally literate, with all patients self-identifying as smartphone-proficient and digitally literate. This sampling characteristic may have positively skewed perceptions of usability and acceptability, and findings may not be generalizable to populations with lower digital literacy or limited smartphone access. Furthermore, technology comfort and technology anxiety were not assessed using validated instruments. Future studies should include such measures, as they may moderate perceptions of usability and acceptability. The survey items were developed for this study and have not undergone formal psychometric validation. Future work should assess their reliability and validity to strengthen measurement rigor. Lastly, the rapid evolution of AI and chatbot tools may outpace the current design.

This pilot study suggests that a clinical protocol–guided chatbot can be feasibly embedded within a hospital tele-support program for T2DM management. Patients reported high usability and trust, and telecarers found chatbot-generated precall information helpful for structuring conversations. However, challenges related to workflow integration and limited content personalization remain key barriers to scaling. While chatbots cannot replace the empathy and relational aspects of human care, they may serve as valuable adjuncts by automating routine assessments and reinforcing education, enabling telecarers to focus on the more complex and relational dimensions of chronic disease management. Future refinements should prioritize smoother workflow integration, greater tailoring to individual needs, and a balanced use of automation to support a meaningful human-digital partnership in care.