This study recruited 192 dentistry students (female: n=98, 51.04%; male: n=94, 48.96%) from a first-class university. In China, the dental medicine curriculum during the first 3 years of university focuses on foundational medical sciences and basic clinical medicine, with the aim of progressively developing clinical reasoning skills. In the fourth and fifth years, students transition to systematic instruction encompassing both theoretical knowledge and clinical skills in core dental disciplines. Therefore, participants in the study were in their fourth or fifth years of a 5-year dental education program and had received theoretical knowledge in courses. They ranged in age from 20 to 25 years, with an average age of 22.53 (SD 1.47) years. All participants’ visual acuity was normal or had been corrected to be normal.

Before the study, participants were calibrated for eye tracking using the 9-point method, and those who did not meet the calibration standards were excluded. Participants who met the inclusion criteria were randomly assigned to a ChatGPT-3.5–assisted learning group and a blank control group using the sealed envelope method to minimize systematic bias. The control group used videos for skill acquisition, while the ChatGPT group supplemented video learning with ChatGPT-3.5 as an additional learning tool. Following the completion of pretest questionnaires assessing theoretical knowledge, spatial ability, motivation, and self-efficacy, participants began a 1-week skill acquisition period. Subsequently, both groups were assessed on their skills using desktop VR (Zhonghui), and eye movement data were collected using an aSee eye-tracking device (EVERLOYAL). Data collection was conducted in a digital classroom with illumination levels maintained between 100 and 130 lux. Participants’ motivation and self-efficacy were also recorded.

Although ChatGPT version 4.0 offers advantages such as enhanced comprehension, reasoning, and accuracy compared with version 3.5, the free version 3.5 (Figure 1) was used in this experiment due to its cost. Given that a recent study indicates nearly all faculty and students in Chinese universities have previous experience with generative AI tools, this study did not assess participants’ previous exposure to ChatGPT [30]. However, standardized operational guidelines were distributed to the ChatGPT group via group messaging platforms before the study to ensure familiarity and proficiency with the technology. During the skill learning period, the ChatGPT group was allowed to use ChatGPT to address doubts that arose during the operation process, verify the correctness of the procedures, and provide additional guidance as needed (Multimedia Appendix 1).

Eye movements were monitored using the aSee eye-tracker (Multimedia Appendix 2) at a sampling rate of 60 Hz, tracking both eyes. Before the experiment, the participant’s position was calibrated to ensure that the distance between the screens and eyes was approximately 70 cm and participants were told they could move their head freely but not too much during the experiment. aSee eye-tracker was applied to collect and analyze the eye movement data.

A comprehensive knowledge test was formulated to assess participants’ theoretical understanding of the surgery (Multimedia Appendix 3). The test comprised 10 multiple-choice questions, and the content validity of these questions had been rigorously examined by experts, ensuring the relevance and appropriateness of the test content, and experts recommended the test be completed in 10 minutes. With each question scored out of 10, the total test score ranges from 0 to 100.

At the end of the experiments, participants were asked to complete an operational test in desktop VR within 15 minutes (Figure 2). Operational test scores are automatically generated by the VR, which avoids the influence of subjective factors on test results.

To assess students’ learning motivation and self-efficacy, this study used questionnaires that have been widely investigated in dental, medical, and nursing practice. The questionnaire contains 5 items focusing on the motivation domain and 5 items focusing on self-efficacy. Each item was coded according to a 5-point Likert rating scale (1=“strongly disagree,” 2=“disagree,” 3=“neutral,” 4=“agree,” and 5=“strongly agree”). The corresponding Cronbach α values were 0.76 and 0.73, indicating satisfactory internal consistency. Validity was censored by experts in medical education. All participants were requested to complete the questionnaire both before and after the experiment, with a time allocation of 10 minutes for this task.

An independent samples t test was used to determine the difference in performance and spatial ability, while the Mann-Whitney U test was used to compare the scores of self-efficacy and learning motivation. The raw pupil diameter data were extracted to measure cognitive load. For further signal processing, data points labeled as a blink and 50 milliseconds before and after these points were removed since eyelid movement during these periods might distort pupil diameter. MATLAB programming (MathWorks) was used to eliminate the fluctuations in pupil size to obtain a smooth curve of pupil size change over time, as depicted in Figure 3. In the analysis of pupil diameter variations during learning, this study calculated the median pupil sizes at the beginning of learning which was marked as the baseline, and the overall median pupil sizes during the learning process separately. The preference for the median over the mean was driven by the former’s greater robustness toward noise and outliers.

The trial protocol followed the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth; version 1.6.1) checklist (Multimedia Appendix 4) [31]. Informed consent was obtained from all participants and the study was approved by the ethics committee of the School and Hospital of Stomatology, Wuhan University (WDKQ2024-034) before it was conducted.

The research results include the distribution of participants, knowledge test scores, operational test scores, and change in pupil diameter, which is used to measure the cognitive load, spatial ability test, learning motivation, and self-efficacy.

During the eye-tracking calibration, 5 participants were excluded because they did not pass through the calibration. Thus, during the eye-tracking calibration, 94 students were randomly assigned to the ChatGPT group and 93 to the control group. The sex distribution was balanced, with 50.80% of female participants (26.2% in the ChatGPT group and 24.6% in the control group) and 49.20% of male participants (24.06% in ChatGPT the group and 25.13% in the control group), ensuring a representative sample. The detailed experimental procedure is illustrated in Figure 4.

This paper presents a comparative analysis between the ChatGPT group and the blank control group in terms of theoretical knowledge and operational performance. Shapiro-Wilk tests were used to assess the normality of the data, which showed that the data in both groups followed a normal distribution. Due to the normality, t tests were performed to determine significant differences.

The results (Table 1) showed that there was no statistically significant difference in the theoretical knowledge test scores between the 2 groups (t₁₈₅=0.649, P=.52), suggesting that the participants’ knowledge levels before learning operations in the 2 groups were comparable. However, the t test for operation performance revealed a statistically significant difference (t_176.241=4.569, P<.001), indicating that the learning modes exerted an impact on operation learning. Specifically, the ChatGPT group exhibited a significant advantage over the video-only group in the operation training. It is noteworthy that all statistical tests were conducted at the α=.05 significance.

To access and compare the cognitive load experienced by the participants during the learning, changes in the participants’ pupil diameters were analyzed. Preliminary screening of the data confirmed that they followed a normal distribution. Figure 5 shows the mean increase in pupil diameter from baseline. An independent samples t test was performed to determine the statistical significance of the changes in pupil size. The results of the t test (P<.001), performed at a significance level of α=.05, showed significant differences between the ChatGPT group (mean 0.137, SD 0.036) and the control group (mean 0.312, SD 0.032). This implies that the cognitive load borne by participants in the ChatGPT group was significantly lower than that in the control group.

In order to categorize the participants according to their spatial abilities, the median was taken as the boundary. Table 2 provides an insight into the distribution and operational performance of participants with varying spatial abilities. In terms of the theoretical knowledge test, no statistically significant differences were found between the two modes, irrespective of whether participants belonged to high (t₉₂=0.689, P=.49) or low (t₉₁=0.764, P=.45) spatial ability groups. The independent samples t test, as presented in Table 2, indicated that there was no compelling evidence to support the notion that learning mode had a different impact on high spatial ability learners, However, for low spatial ability learners, the t test results indicated a statistically significant impact of the learning mode. With regard to those with low spatial abilities, the operational scores achieved by the ChatGPT group (mean 70.20, SD 10.71) were found to be higher than those attained by the control group (mean 55.41, SD 13.31).

Given that learning motivation and self-efficacy scores were not normally distributed, the Mann-Whitney U test was used to analyze the differences between the pretest and posttest scores for ChatGPT and the control group respectively. Furthermore, this study further compared the scores between the 2 groups. Figure 6 demonstrates that, at the outset of the study, there was no evidence to suggest that there were any differences in learning motivation (Z=0.31, P=.76) and self-efficacy (Z=0.69, P=.48) between the 2 groups before the intervention. Nevertheless, it is notable that the posttest scores for learning motivation and self-efficacy differed from the pretest scores, irrespective of whether the learners were in the ChatGPT or control group. Furthermore, in comparison to the control group, the ChatGPT group exhibited higher posttest scores for both learning motivation (Z=2.32, P=.02) and self-efficacy (Z=2.03, P=.04).

The study revealed that incorporating ChatGPT-3.5 as a tutor resulted in markedly elevated performance scores in skill assessments compared with using videos in isolation. This discrepancy was particularly pronounced among students with lower spatial abilities. In addition, the use of ChatGPT-3.5 was shown to reduce cognitive load, enhance self-efficacy, and boost learning motivation. These findings provide robust evidence supporting the use of ChatGPT-3.5 as a valuable tool in skill training and offer insights for the design of educational programs.

Our investigation builds on previous studies that have identified the potential of ChatGPT in medical education. A recent study demonstrated that ChatGPT exhibited exemplary performance in both the immediate and long-term contexts of orthopedic teaching for undergraduate students [1]. In the field of dental education, extant research indicates that ChatGPT demonstrates satisfactory performance across a range of dental assessment types [32,33]. However, these studies have primarily focused on theoretical knowledge. Our findings further reveal that ChatGPT exhibits comparable exciting potential in skills-based education. Compared with traditional dental education, ChatGPT offers personalized learning content and feedback tailored to individual student needs, helping students acquire skills more effectively. In addition, ChatGPT provides real-time feedback and answers to student queries, thereby enhancing the efficiency of the learning process [34]. It also alleviates the strain on teaching resources and offers greater flexibility in skill training schedules.

Changes in pupil diameter during task performance have been used to infer variations in cognitive load. A previous study applied this method to measure cognitive load differences between experts and students while examining dental radiographs [35]. In line with this approach, this study used eye-tracking technology to record pupil diameter changes from baseline. The results indicated that participants in the ChatGPT-assisted group exhibited smaller changes in pupil diameter, suggesting a reduction in cognitive load. This decrease can be attributed to ChatGPT’s ability to address challenging points in the skill-learning process. These findings not only validate the efficacy of ChatGPT in reducing cognitive load but also highlight its potential as a pedagogical tool that fosters a smoother and more relaxed learning experience.

Emerging evidence indicates the varied performance of AI models in spatial relations–related topics. Previous studies evaluating ChatGPT on Geographic Information Systems examinations demonstrated its capacity to achieve passing scores in spatial analysis, spatial statistics, and interpolation tasks [36]. Furthermore, generative AI systems including ChatGPT-3.5 have shown nascent potential in executing basic spatial queries [37]. However, limitations persist, as evidenced by its suboptimal performance in robot programming scenarios requiring complex 3D spatial reasoning and nuanced understanding of spatial relationships [38]. Nevertheless, there is no research on the topic of the spatial relations related topics between ChatGPT and medical skill learning, and this study provides a preliminary exploration of this component.

Previous research has demonstrated a significant yet modest positive correlation between spatial ability measured by the Purdue Spatial Visualization Test: Rotations and performance in dental anatomy assessments [39]. Furthermore, studies using other spatial ability assessments, such as the mental rotations test, visualization of views test, and visualization of rotation test, have consistently shown that students with higher spatial aptitude achieve superior performance in endodontics [40], radiology [41], anatomy [42], and prosthodontics [43]. The convergent validity of these findings across diverse spatial ability metrics and dental subdisciplines suggests that spatial ability may serve as a foundational competency in skill acquisition. This study delves into the impact of spatial ability on learning outcomes under different instructional modes, revealing the interaction between spatial ability and teaching methods. The findings indicate that ChatGPT-assisted instruction significantly enhances learning outcomes for learners with lower spatial ability. However, no significant difference was observed between the 2 instructional modes for learners with higher spatial ability. Cognitive load theory provides a framework for understanding these results. Given the limited capacity of working memory to process information simultaneously, learners with lower spatial ability in the control group experienced cognitive load beyond their cognitive resources [20]. In contrast, learners with higher spatial ability could activate preconstructed schemas based on 2D images, thus reducing the demand on working memory [20,44]. ChatGPT, by offering clear explanations and guidance, helps learners with lower spatial ability better comprehend complex skills or concepts, thereby alleviating cognitive load and improving their learning outcomes.

Besides, the findings of this study indicate that participants in both learning modes experienced improvements in learning motivation and self-efficacy, with ChatGPT-assisted learning demonstrating a particularly pronounced capacity to facilitate these outcomes. Previous studies also support this view [45]. A thematic analysis of interviews with higher education experts indicated that the personalized feedback and support provided by ChatGPT can assist students in setting and achieving goals, reflecting on their progress, and enhancing noncognitive skills such as motivation and self-efficacy [46,47].

As an auxiliary tool in medical education, ChatGPT is currently considered a double-edged sword by many scholars [3,48]. On one hand, it may impede the development of students’ critical thinking and independent learning abilities and potentially encourage academic dishonesty [3,49-52]. However, when applied to skill training, ChatGPT displays considerable promise. It is capable of providing tailored assistance and feedback in response to the learner’s progress, offering technical guidance and confirmation of procedural steps [53]. This aids in the comprehension of complex skills, such as tooth preparation, periodontal scaling, and impacted tooth extraction [7]. Furthermore, it diminishes cognitive load during the learning process, enhancing both learning motivation and self-efficacy. Therefore, dental educators must not choke on their knowledge but rather set a new standard for teaching methods and assessment to keep up with the times.

First, it is important to recognize that ChatGPT has some inherent limitation types, although ChatGPT is trained on big data, there is a possibility that the training data may be inherently biased or there may be errors in the training process, which leads to the accuracy and reliability of the information provided will also be affected [54]. During the skill learning process, individuals will have their distinctive queries, so for this study checking the accuracy of the answers to the ChatGPT responses was difficult. Second, this study was conducted using the free version of ChatGPT-3.5, and more research is needed to explore whether ChatGPT-4.0 would be more advantageous in aiding skill instruction [55]. Besides, the study compares ChatGPT with a blank control group based on videos, and further research is required to investigate the detailed strengths and weaknesses of ChatGPT in comparison to other methods.

This study sought to ascertain whether the supplementary use of ChatGPT-3.5 is more effective for mastering basic oral skills operations among dental students compared with videos alone in terms of manipulative performance, cognitive load, spatial ability, and emotions. The study used desktop VR for skill testing and the results showed that students with the assistance of ChatGPT performed better. Eye-tracking technology was used to record the visual behaviors, and the results revealed that participants in the ChatGPT-3.5 group experienced reduced cognitive load. The interaction analysis highlighted learners with low spatial ability derived greater benefits from the ChatGPT than those with high spatial ability. In addition, the questionnaire revealed that learners in the ChatGPT group demonstrated higher levels of self-efficacy and learning motivation. In conclusion, the findings of this study contribute to recognizing the potential of ChatGPT in dental skills education.