We conducted a prospective 2-arm randomized controlled trial with a pretest-posttest study design. Social media platforms were used to recruit participants who were undertaking year 3 of their nursing courses in a local university. We used the rule of thumb for a pilot 2-arm trial sample size involving at least 55 participants [23]. Accounting for a 10% overall dropout rate, the total sample size was planned to be 64 participants. A total of 67 participants expressed their interest in the study. After screening for eligibility and obtaining written informed consent, the study coordinator randomized 65 participants using a web-based random number generator to either AI-powered or human-controlled groups [Multimedia Appendix 1].

This study was approved by the National University of Singapore institutional review board (ref no. NUS-IRB-2022-202). Before the study intervention, both groups met the research team through a Zoom videoconference to receive information about the study and provide their written consent. They were assured that participation was entirely voluntary, and withdrawal would not affect their academic performance.

Participants in both the AI-powered and human-controlled groups were scheduled to participate in a 2-hour VRS on sepsis team training remotely. The participants in the AI-powered group were scheduled to undertake the VRS individually while being supported by the research team via Zoom chat for any enquiries. Participants in the human-controlled group were assigned to groups of 4 to 6 team players to engage in the VRS with medical students and facilitators. The design and development of both forms of VRS, which were grounded in experiential learning theory, has been described and evaluated in previous studies [18,22].

In both study interventions, participants were involved in 2 simulation scenarios. The first scenario simulated a morning medical round, involving a postoperative patient with early manifestations of sepsis, which required early goal-directed management of sepsis. The second scenario involved the same patient whose condition had deteriorated into septic shock and required airway management and fluid resuscitation. The participants were given time to read the case history before commencing each scenario. Both scenarios began with a nursing participant performing nursing assessment and management of the virtual patient using the ABCDE (Airway, Breathing, Circulation, Disability, Expose) approach, followed by communicating with an AI virtual doctor or a doctor avatar controlled by the medical student. A voice chatbot learning system was built using Google Cloud’s Dialogflow engine to train the AI doctor, which operates through deep neural networks to recognize and predict human-agent conversation patterns [21]. The AI doctor’s responses were modeled based on gathered conversational data set between nursing and medical students from previous studies [18,20].

Figure 1 illustrates the participants’ viewpoints when interacting with the AI-powered doctor or the medical-student-controlled avatar. We adapted communication strategies from the TeamSTEPPS (Team Strategies and Tools to Enhance Performance and Patient Safety) curriculum, which included ISBAR (Identity, Situation, Background, Assessment and Recommendation) and CUS (Concerned, Uncomfortable and Safety) feedback to acknowledge, call out, and check back [24]. In these scenarios, the nursing participants were expected to use communication strategies to communicate the assessment findings to the AI-powered doctor or human-controlled doctor avatar.

Each simulation scenario lasted 15 to 20 minutes and was followed by a debriefing to enable participants to receive feedback on their performance regarding the assessment and management of sepsis and septic shock and their interprofessional communication. As illustrated in Figure 1, the AI-powered group received a self-directed debrief checklist to review their performance, whereas the human-controlled group engaged in facilitator-led debriefing. In both group debriefings, the ABCDE and TeamSTEPPS communication tools provided the frameworks to guide learning through feedback.

We administered the knowledge tests and self-efficacy scale before (baseline) and immediately after (posttest) interventions for both groups. The 8-item communication knowledge and 18-item sepsis knowledge tests were developed and content validated by a multidisciplinary team comprising a medical doctor, an advanced practice nurse, and nursing academics. This study reported a high Cronbach α of .81. The Patient Clinical Information Exchange and Interprofessional Communication Self-Efficacy Scale, a 6-item questionnaire using a 0-100 Likert scale developed by Hernández-Padilla et al [25], was used to measure participants’ perceptions of self-efficacy in team communication based on the ISBAR communication strategy. A high internal consistency with a Cronbach α of .93 was obtained for the Patient Clinical Information Exchange and Interprofessional Communication Self-Efficacy Scale in this study.

Participants from both groups were scheduled to undertake simulation-based assessment within 2 weeks of postintervention to determine their sepsis care and interprofessional communication performance. Participants were given a case history to read and an orientation of the simulation room with a manikin setup. The simulation-based assessment involved the participants performing the nursing assessment and management of the manikin, which displayed signs and symptoms of deterioration, and communicating with the doctor to provide team care. Each simulation assessment lasted approximately 15 minutes, and the entire process was recorded. The recorded videos were sent for rating by 2 assessors, who were blinded to the groupings. The assessors rated the communication and sepsis care performances independently using a validated team communication scale and the RAPIDS (Rescuing A Patient In Deteriorating Situation) tool. The team communication scale comprising a 9-item checklist was developed by the research team based on observable nurse-doctor communication using the TeamSTEPPS communication strategies. The scale was content validated by an interprofessional team of 5 nursing academics and clinicians. The RAPIDS tool was adopted from a previous study to measure nurses’ simulation performance in assessing and managing a deteriorating patient [26]. Interrater reliability across the 2 assessors was computed based on their independent scoring on the video-recorded performances using the validated team communication scale and RAPIDS tool. An overall κ value of 0.832 was reported, indicating good interrater agreement.

We applied descriptive statistics, chi-square tests and t tests, to analyze the demographic characteristics of the study population. We computed the paired sample t test to examine significant changes between the baseline and posttest performance scores and used ANOVA to determine differences in the posttest scores between the groups. The level of statistical significance was set at P<.05.

Sixty-four nursing students were recruited in the study. The majority were women (50/64, 78%) and Chinese (51/64, 80%) and had a mean age of 22.2 (SD 2.20) years. No significant differences were observed in the baseline characteristics, including age (P=.49), gender (P=.55), and ethnicity (P=.54) between the AI doctor and human doctor groups (see Table 1). This supported the homogeneity of participants between the 2 groups.

The simulation-based assessment revealed no significant differences in the sepsis care performance scores between the AI-powered and human-controlled groups (F_1,62=0.75, P=.39, η²=0.012), though a higher mean score was observed in the AI-powered group (see Table 2). In contrast, significant differences were noted in the sepsis care knowledge scores between the participants in the AI-powered and human-controlled groups (F_1,62=7.37, P=.009, η²=0.106). As shown in Table 2, the AI-powered group reported higher sepsis care knowledge mean scores (mean 9.06, SD 1.78) than the human-controlled group (mean 7.75, SD 2.08). Within-group comparison demonstrated a significant increase in sepsis care knowledge from the baseline in the AI-powered group (t₃₁=−5.21, P<.001) but not the human-controlled (t₃₁=−1.43, P=.16) group.

As presented in Table 2, within-group comparison revealed significant improvements in communication knowledge and self-efficacy from baseline levels in both AI-powered and human-controlled groups. Although between-groups comparison showed no significant differences in communication knowledge scores (F_1,62=0.514, P=.48, η²=0.008) and communication performance scores (F_1,62=1.62, P=.21, η²=0.026), the human-controlled group reported higher communication self-efficacy mean scores (mean 69.6, SD 14.4) than the AI-powered group (mean 60.1, SD 13.3).

To the best of our knowledge, this is one of the first studies to evaluate the effectiveness of AI-powered VRS by comparing it with human-controlled VRS. This randomized controlled trial study did not demonstrate any significant differences in sepsis performance and interprofessional communication performance between participants in the AI-powered and human-controlled groups. However, we found significant differences in sepsis care knowledge and communication self-efficacy between the 2 groups. Although the AI-powered group obtained significantly higher sepsis care knowledge scores than the human-controlled group, the human-controlled group reported significantly higher self-efficacy in interprofessional communication than in the AI-powered group. The underlying learning principles and theories that guided the use of AI and human-controlled VRS could be applied to explain these findings.

The simulation-based assessment performance outcomes did not reveal inferiority of AI-powered VRS regarding either sepsis care or communication skills performance when compared with human-controlled VRS. Despite the different medical team virtual player agents, both VRS approaches offered experiential learning and cognitive tools that allowed the nursing students to practice the assessment and management of sepsis and team communication skills through role playing and reflection. An earlier study demonstrated the need for both cognitive tools and experiential learning modalities to support the development of a shared mental model and optimal teamwork delivery [27]. The application of experiential learning using various simulation modalities, including computer-based simulation and manikin-based simulation, has demonstrated positive outcomes in clinical performance and team communication related to the care of patients with clinical deterioration [28,29]. Besides grounding the study in experiential learning theory, we also ensured close alignment between the simulation task and the clinical task in both VRS groups. Thus, findings from this study provided further evidence to support Hamstra et al’s [30] recommendation of focusing on high functional fidelity rather than physical fidelity of simulation to develop desired performance outcomes.

Interestingly, our findings demonstrated that the AI-powered group had significantly higher sepsis knowledge scores than the human-controlled group. This finding supported our earlier qualitative data in which the AI-powered doctor was perceived by the nursing students as a more knowledgeable agent than the doctor agent controlled by medical students [22]. Although both AI- and human-controlled VRS approaches were based on experiential learning, they used different approaches to support experiential learning. Kiili’s [31] experiential gaming model was applied to build the game design of AI-powered VRS, which involved multiple quizzes as a form of challenge for learners to interpret and problem-solve patient assessment data with an awarded point system. The AI-powered virtual agent was also designed to involve learners in a reasoning process through questioning. The importance of using the AI-powered virtual agent to facilitate knowledge construction, rather than to provide knowledge, has been highlighted in several studies; this approach ensures the development of learners’ reasoning process for knowledge construction [32,33]. Using a self-regulated learning approach, feedback was delivered through a checklist format with evidence-based decision-making rationales. With growing attention given to the application of AI to support assessment, AI-driven learning analytics for assessment can be embedded into the VRS to generate feedback to students, as well as to provide informative data for educators to track learners’ learning outcomes [34].

Although no significant difference was noted regarding interprofessional communication performance, the human-controlled group reported a significantly higher level of self-efficacy for interprofessional communication than in the AI-powered group. Unlike AI-powered groups, participants in the human-controlled group were given the opportunity to engage in social interactions among learners through role playing and debriefing. The theory of social constructivism has emphasized the importance of learning from social interaction [35], which has been predominantly applied in multiuser virtual worlds to underpin learning activities [36]. Thus, social constructivism could be applied to explain how our human-controlled multiuser VRS might bring about higher self-efficacy of interprofessional communication than in the AI-powered simulation. Although we have been working toward improving the AI-powered doctor’s affective states consisting of facial expressions and comprehension of natural conversation, we acknowledged that the fidelity and authenticity of our AI-powered agent poses challenges in promoting social interaction through the human-AI conversation. This suggested the need for further innovative development to drive a more sociably enabled AI using emotion-expressive virtual agents, which can be realized by seeking progression in multimodal computing and machine learning [37,38].

The lack of substantial differences in performance between AI and human-controlled VRS highlighted the potential role of AI-powered VRS in supporting nursing students with interprofessional training, particularly in circumstances when the accessibility and availability of medical students are lacking. In addition, variations in learning outcomes between the groups suggested the benefits of combining the different simulation modalities to provide an optimal learning approach. Our previous study recommended the use of scaffolding for the instructional sequence of interprofessional learning activities within blended learning environments [39]. Thus, a blended learning approach commencing with concept building using AI-powered VRS, followed by experiential learning with medical students in a virtual environment, and subsequently using face-to-face simulation-based interprofessional learning could be implemented and evaluated in future studies.

The limitations that warrant attention are few. First, we acknowledged the variations in experiential learning approach between the 2 simulation modalities as confounding variables. However, the performance outcome measurements used in this study were closely aligned with the learning objectives. Second, despite evaluating participants’ performance using simulation-based assessment and a validated tool, this was limited by an immediate posttest performance in the simulation setting. Third, similar to the performance test, we did not measure the long-term retention of knowledge and level of self-efficacy. Thus, future studies could evaluate learning outcomes over a longer period and measure the impact in the clinical setting. Finally, the effectiveness of AI-powered VRS was not optimized by allowing participants to have more than one exposure as the intent was to enable deliberate practice opportunities.

An AI virtual doctor agent was embedded in a virtual environment to engage interprofessional team training of nursing students on sepsis care. The performance outcomes from simulation-based assessment did not suggest that the AI-powered VRS provided inferior sepsis care and interprofessional communication training when compared with a human-controlled VRS. This shed light on the effectiveness of AI-powered medical team players in supporting nursing students with interprofessional learning where the opportunity to form interprofessional teams with medical students is lacking. Our findings suggested the need for further innovative development in AI-powered VRS to promote social connectedness with learners and support AI-driven learning analytics for assessment. Given the varied learning outcomes between AI-powered and human-controlled VRS approaches, our study recommended blending them along with face-to-face simulation to optimize students’ performance in sepsis care and interprofessional communication.