This study follows scoping review logic and procedures [19] and reports according to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) and PRISMA-S (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Literature Search Extension) guidelines [20,21] (Checklist 1). This approach is suitable for analyzing a range of studies to describe and map the overall picture of the current scope of the evidence in the field [22]. Given the nature of the study design, ethical approval was not required. Similarly, as per scoping review methodology, we did not formally assess the quality of the included studies [23]. The protocol for the study has not been published or formally registered. An operational workflow, discussed as a team, guided the conduct of the review; consistent with the exploratory and mapping-oriented nature of scoping reviews, aspects of the approach were refined as our understanding of the literature developed. These adaptations, described in the following section, relate primarily to the analytical framework.

The population-concept-context framework was defined as follows: the population comprised undergraduate medical students, the concept was the use of GenAI in medical education, and the context was education-related empirical research (Multimedia Appendix 1). We developed and refined the search strategy through iterative testing. An electronic search was conducted between September 22 and September 24, 2025, across Ovid (MEDLINE and Embase), PubMed, Web of Science, and Scopus to maximize coverage. Free-text terms ([“GenAI” OR “generative AI” OR “large language model” OR “gpt” OR “deepseek” OR “LLM”] AND [“medical student” OR “medical education”]) aimed at maximizing coverage were applied to titles and abstracts, with no language or date restrictions.

The search yielded 2316 records. After deduplication, we used TERA (The Evidence Research Accelerator) tools [31] to screen 923 records according to inclusion and exclusion criteria that were thoroughly discussed and agreed upon as a team. Our a priori exclusion criteria comprised (1) reviews, commentaries, letters, and editorials; (2) preprints, conference papers, and study protocols; and (3) studies that did not pertain to undergraduate medical education (ie, postgraduate medical education—residency or continuing professional development; we allowed studies with mixed populations, provided that the population of interest was included). During title and abstract screening, it became apparent that many records concerned the technical performance of GenAI or GenAI use without student-facing interventions. Because the Kirkpatrick model is designed to evaluate educational interventions, it would not be possible to extract relevant information and evaluate these studies within the chosen model. Hence, we refined and tightened the eligibility criteria to include only student-facing intervention studies. As a result, 2 categories of studies were excluded from the final analysis. First, studies evaluating GenAI performance in medical examinations without student participants were excluded, as they focused solely on whether GenAI could simulate student performance. Second, we excluded studies on GenAI in medical education that did not include a student-facing intervention-based study focused on educational student outcomes, for example, studies where content generation was evaluated, or students’ awareness of GenAI tools was studied.

During the title and abstract screening, each researcher independently screened their assigned set of records; we then reached an agreement in pairs (records numbered 1‐462: AP and ON; 463‐923: SPH and JH). To ensure consistent application of the eligibility criteria, the authors (AP, ON, SPH, and JH) met on multiple occasions to discuss the screening approach. We also conducted a dedicated in-person meeting to jointly review a sample of 40 records to test the consistency of the shared interpretation of the eligibility criteria. The team determined that the level of agreement was sufficient to proceed; however, agreement statistics were not formally recorded. This process resulted in 269 eligible records. The tightened inclusion criteria yielded 45 studies to be sought for retrieval. After full-text screening, the final dataset comprised 27 records.

Data were extracted in pairs using a collaboratively developed and piloted template and recorded in Microsoft Excel. Following full-text analysis, extracted data were reviewed and finalized through team consensus. Unstructured data were standardized to enable quantitative synthesis, and descriptive statistics were used to summarize key characteristics as well as the Kirkpatrick classification. Data extraction included country, aim, study design, use (purpose) of the GenAI tool, data collection, results and outcomes, reference to pedagogical theory, the Kirkpatrick level, and notes. We allowed for nonexclusive categorization across Kirkpatrick levels; that is, individual studies could be classified under more than one level.

In light of the use of AI for textual data coding observed in our dataset, we needed to take a clear position on how such studies should be classified and what constitutes qualitative analysis. Our classification reflects a stance rooted in qualitative research traditions, in which coding is understood as the beginning rather than the endpoint of qualitative analysis [32,33]. We considered studies to be qualitative or mixed methods only if they demonstrated core qualitative commitments such as interpretive labor, reflexive engagement, contextual meaning-making, and an accountable analytic narrative. In our view, combining descriptive qualitative analysis focused primarily on the summarization and quantification—which already entails limited interpretive accountability—with automated, nonhuman analytic judgment does not demonstrate the interpretive and accountable meaning-making that characterizes qualitative inquiry. We do not dismiss the value of automated topic classification methods; however, none of the studies that relied on semiautomated AI coding in our dataset provided evidence that such interpretive effort had been undertaken; hence, we classified them as quantitative, even if they self-identified as mixed methods, which may be recognized as representing a rather conservative methodological stance [34] within contemporary discussions on automating qualitative research [35,36].

As a review of previously published literature, this study did not require ethical approval.

Twenty-seven studies were included in this review (Figure 1), the majority of which (n=21) were published in 2025, with a substantial proportion appearing in the Journal of Medical Internet Research and its sister journals (n=9) or in BMC Medical Education (n=4). Most studies were conducted in the United States (n=6) and China (n=5), with the remainder from Germany, Japan, Mexico, Sweden, Colombia, France, India, Italy, Turkey, and the United Kingdom. Most studies used quantitative designs (n=23), with 4 studies (2 mixed methods and 2 qualitative studies) [37-40] that used AI to automate qualitative analysis and were therefore not classified as mixed methods.

GenAI was used for 3 purposes: simulated patient experience (n=13), knowledge or decision support (n=10), or educational content generation (n=4). Most studies (n=25) used OpenAI’s ChatGPT either as a standalone tool (n=17) or integrated within other platforms or multitool workflows (n=8). Other standalone tools included DeepSeek and Kimi Chat. In a few cases, GenAI tools were used in conjunction with other digital tools, including Stable Diffusion [37,38], DALL·E 3 [41], Leonardo.ai [37], and Whisper-3 [39]. Two studies [37,38] presented particularly sophisticated workflows incorporating tools for plot, image, narration, and audio generation to create videos.

Of the 27 included studies, 14 were classified within a single Kirkpatrick category, whereas 13 spanned multiple levels. Ten studies were positioned across levels 1 and 2, examining both learner reactions and learning outcomes [38,49-57]. Two studies spanned levels 1, 2, and 4 [30,41], and one extended further to include the additional “how” category [61], representing the most conceptually comprehensive examination of educational impact.

Overall, the Kirkpatrick-focused analysis revealed a strong concentration of studies on learner reactions and learning outcomes, with 13 spanning across levels. Beyond the Kirkpatrick framework, a substantial body of literature fell outside educational intervention evaluation altogether: 119 studies did not report on student-facing interventions, and a further 110 focused exclusively on GenAI performance in assessments or examinations. Studies at level 1 consistently reported positive learner perceptions and satisfaction with GenAI-supported tools, while level 2 studies generally suggested improvements in learning outcomes. Notably, very few studies examined how GenAI impacted learners or were theoretically grounded.

The predominance of studies reporting outcomes at the reaction and learning levels, with few addressing behavioral or system-level outcomes (levels 3 and 4), should not be viewed as an anomaly. Similar patterns have been documented not only in GenAI research [16,18], but also across studies of other educational technologies, such as virtual simulation [63], and instructional approaches including team-based learning [64]. Prior reviews suggest that evidence at levels 3 and 4 tends to emerge when technologies are examined over extended periods—for example, in a study of web-based continuing medical education [65], technology-enhanced oncology education over a decade [66], or e-learning in undergraduate clinical education over two decades [67]. These observations indicate that levels 3 and 4 outcomes may depend on the duration and extent of technology integration. Given the cyclical emergence of new educational technologies, with attention frequently shifting to successive innovations [2], the low prominence or absence of levels 3 and 4 outcomes, understood as outcomes in clinical settings, is therefore expected; however, in light of the broadened definitions introduced in this review (sustained behavior for level 3 and theoretical grounding for level 4), their low representation points toward a deeper issue in how the field designs and studies educational interventions.

Despite being situated within medical education, a substantial proportion of studies displace students from the center of inquiry in 2 key ways. First, there is limited use of explicit theoretical grounding; second, many studies focus on what GenAI can do for educators or how it performs. The marginalization of medical students is not new. Ludmerer [68] described a similar dynamic resulting from the expansion of biomedical research and clinical service, which produced what he termed “the forgotten medical student.”

In our analysis, we draw attention to the volume of studies that prioritize tool performance over learning processes, thereby directing attention away from how students learn with, through, or alongside GenAI. In doing so, GenAI research risks repeating historical cycles of educational technology enthusiasm without generating durable pedagogical insight. Our findings mirror those of Aster et al [69], who reported that 52% of studies focused primarily on demonstrating GenAI tool performance rather than educational impact. Similarly, the limited presence of explicit theoretical grounding reflects broader trends in medical education research [18,29], despite theory’s importance in explaining why technology-enabled interventions succeed or fail [29]. A related concern is the tendency to treat Kirkpatrick levels as disconnected evaluative categories rather than interconnected elements of educational design. Few reviews report studies spanning multiple levels [70]. These findings reinforce fragmentation across design, theory, and evaluation, as well as insufficient foregrounding of students at the outset of research design. We therefore argue that the key issue is not insufficient outcomes but insufficient integration across design, theory, and evaluation, resulting in fragmentation rather than integration of steps, which results in the systemic marginalization of students as analytic participants.

We therefore propose the Technology-Integrated Educational Design (TIED) road map, a fresh look at the Kirkpatrick model that encourages the integration of steps across educational design, implementation, and evaluation (rather than treating levels as discrete categories for evaluation purposes only), while also showing the often-overlooked preinstructional stages of educational innovation. The TIED road map is intended as a reminder to reposition students at the center of the intervention lifecycle (Figure 3).

An integrated design begins with preinstructional exploration, addressing questions of trustworthiness, appropriateness, and learner needs: Is the tool suitable for educational use? How can it meaningfully support learning? What needs will it meet? This exploratory phase anticipates downstream learning, behavioral, and system-level implications and is grounded in theoretical considerations from the outset. Subsequent stages involve examining learner responses; evaluating changes in knowledge, skills, and attitudes; and assessing transfer into sustained practice. Crucially, the logic also requires mechanism-oriented inquiry that examines how GenAI functions pedagogically—what works, for whom, and under what conditions.

Rather than sequential checkpoints, these steps should inform design, implementation, and evaluation continuously and iteratively. Without such integration, research risks reproducing fragmented, single-level studies that prioritize tool performance or isolated outcomes while marginalizing students as active participants in learning. The TIED road map serves as a reminder to maintain student centrality and support cumulative, theory-informed knowledge development in GenAI-enabled medical education.

The TIED road map has not been conceived in a vacuum but builds on previous work in educational evaluation, instructional design, and user-centered digital innovation. We believe its contributions lie in 4 main areas. First, it extends the New World Kirkpatrick Model [14], which advocates working backwards from desired system-level outcomes. In doing so, it reinforces the original intention of the New World Kirkpatrick Model by encouraging its use not only as an evaluation framework but also as a design framework. Moreover, the TIED road map adopts this principle of backward design while concurrently redefining levels 3 and 4 (Evaluate stages for System and Practice) for undergraduate medical education. In doing so, it encourages educators and researchers to reflect early in the design process on what learner behaviors enacted consistently would look like and how these behaviors might contribute to broader pedagogical impact.

Second, TIED explicitly incorporates mechanism-oriented thinking. The road map responds to previous critiques of the Kirkpatrick model, which have argued that it offers limited guidance on evaluating intervention mechanisms [24,25]. TIED explicitly incorporates a mechanism-oriented step, encouraging early and iterative reflection on how interventions produce educational outcomes.

Third, TIED makes the preinstructional phase explicit. The Explore stage aligns closely with human-centered design approaches in [71] by emphasizing the exploration of learner needs alongside experimentation with the capabilities, trustworthiness, and appropriateness of technology. Similarly, the Listen stage reflects formative usability and user experience evaluation and parallels digital innovation frameworks, such as the IDEAS (integrate, design, assess, and share) framework for digital interventions, which emphasize iterative user feedback and refinement throughout the design process [72]. Rather than treating these activities as separate design exercises, TIED incorporates them into the educational intervention lifecycle, explicitly linking educational design, educational evaluation, and human-computer interaction.

Finally, the principal contribution of TIED lies in integrating these perspectives within a single road map. As Susan Gasson argued, user-centered systems development can fail to promote human interests when it becomes dominated by technical goals rather than the people it is intended to serve [73]. Similarly, educational technology research may risk treating needs assessment, intervention design, implementation, mechanisms of learning, and evaluation as isolated activities. By bringing these elements together within a single iterative road map, TIED positions students—not the technology or the evaluation metrics—as the central reference point throughout the educational intervention lifecycle.

This review has several strengths, alongside important limitations. Given the rapid pace of publication in GenAI, additional relevant studies may have emerged after the September 2025 search, limiting the comprehensiveness of the evidence mapped in this review. We also acknowledge that expanding the scope of the review beyond undergraduate medical education to include graduate, in-training, and other professional education contexts could have provided a more robust and comparative analysis.

Anticipating critiques aligned with Yardley and Dornan’s [27] argument that the Kirkpatrick framework rests on assumptions best suited to relatively simple instructional designs and short-term outcomes, we acknowledge that alternative analytical approaches could yield more action-oriented recommendations for educators and researchers. However, given that the field of GenAI in medical education remains in an early, exploratory phase, we deliberately focused on identifying the most salient characteristics and dominant trends within the existing scholarship. Our aim was not to prescribe detailed suggestions for practice but to provide a foundation for broader reflection and to encourage a shift in prevailing research orientations.

This review demonstrates that research on GenAI in medical education is currently concentrated on learner reactions, learning outcomes, and tool performance, with limited attention to learner-centered processes and mechanisms. We argue that there is a pressing need for integrated educational design that connects preinstructional exploration, learning processes and theory, and evaluative considerations across stages. We propose the TIED road map, which builds on established work in educational evaluation, instructional design, and human-centered digital innovation. By integrating preinstructional exploration, mechanism-oriented thinking, and backward design with redefined levels 3 and 4 within a single student-centered framework, the road map encourages educators and researchers to consider not only whether technology-enabled interventions work but also how, for whom, and under what conditions they support meaningful learning.