This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Technology-enhanced learning (TEL) programs are increasingly seen as the way in which education for health care professionals can be transformed, giving access to effective ongoing learning and training even where time or geographical barriers exist. Given the increasing emphasis on this mode of educational support for health care practitioners, it is vital that we can effectively evaluate and measure impact to ensure that TEL programs are effective and fit for purpose. This paper examines the current evidence base for the first time, in relation to the evaluation of TEL programs for health care professionals.
We conducted a systematic review of the current literature relating to the evaluation of TEL programs for health care professionals and critically appraised the quality of the studies.
This review employed specific search criteria to identify research studies that included evaluation of TEL for health care professionals. The databases searched included Medline Ovid, Cumulative Index of Nursing and Allied Health Literature Plus Advanced, Applied Social Sciences Index and Abstracts, ZETOC, Institute of Electrical and Electronics Engineers Explore Digital Library, Allied and Complementary Medicine, and Education Resources Information Center between January 2006 and January 2017. An additional hand search for relevant articles from reference lists was undertaken. Each of the studies identified was critically appraised for quality using the Crowe Critical Appraisal Tool. This approach produced a percentage total score for each study across specified categories. A proportion of the studies were independently assessed by an additional two reviewers.
The review identified 21 studies that met the inclusion criteria. The studies included scored totals across eight categories within a range of 37%-95% and an average score of 68%. Studies that measured TEL using learner satisfaction surveys, or combined pretest and posttest knowledge score testing with learner satisfaction surveys, were found to be the most common types of TEL evaluations evident in the literature. The studies reviewed had low scores across reporting on ethical matters, design, and data collection categories.
There continues to be a need to develop effective and standard TEL evaluation tools, and good quality studies that describe effective evaluation of TEL education for health care professionals. Studies often fail to provide sufficient detail to support transferability or direct future TEL health care education programs.
The term technology-enhanced learning (TEL) is often used to describe a broad field of digital technologies used to support and mediate educational activities [
A range of published reports have highlighted TEL as an effective method to support health care education [
The general availability of mobile and flexible technologies enables learners to minimize time away from health care settings to undertake training and to engage with learning resources when and where they are most suitable to their needs [
The ability to demonstrate the added value and impact of TEL for health care education remains challenging. Previous authors have captured the nature of the challenges in the review and evaluation of TEL within medical education [
A range of studies exist in the TEL literature which document implementation of TEL within medical and health care educational approaches. However, studies demonstrating a comprehensive TEL evaluation or use of standardized TEL evaluation tools in practice are fewer in number. Previous authors such as Ellaway [
This systematic review of the literature aimed to identify studies that have implemented TEL evaluation for CPD and postgraduate or work-based TEL health care education programs, and to assess these using a published critical appraisal tool. The studies identified provide an evidence base for the evaluation and development of future TEL programs for health care professionals.
The review was carried out using a systematic integrative review method. This method allows for the inclusion of empirical and theoretical literature and quantitative and qualitative studies. This method enabled an increased number of studies to be included in the review and is appropriate for the review of evidence to highlight gaps in the literature [
The inclusion and exclusion criteria for this review were developed using the Population, Intervention, Comparison, and Outcomes (PICOS) [
The following electronic databases were searched: Medline Ovid, Cumulative Index of Nursing and Allied Health Literature (CINAHL) Plus Advanced, Applied Social Sciences Index and Abstracts (ASSIA), ZETOC, Institute of Electrical and Electronics Engineers (IEEE) Explore Digital Library, Allied and Complementary Medicine (AMED), and Education Resources Information Center (ERIC) between January 2006 and January 2017 (see
Application of the Population, Intervention, Comparison, and Outcomes (PICOS) Framework to the research question. TEL: technology-enhanced learning.
Parameter | Details |
Participants | Health care professionals in full-time or part-time employment undertaking continuing professional development that is delivered using TEL (full time higher and further education students, school learners excluded). |
Intervention | Studies using a TEL evaluation tool or framework to evaluate technology-enhanced health care education programs. The evaluation tool or framework must be used to evaluate a program for health care professionals. |
Comparison | Some studies will have no comparison or comparator; others will examine one type of TEL approach against another. |
Outcomes | Study must include: (1) evaluation of effective use of TEL, (2) the techniques being evaluated must be sufficiently specified, (3) assessment of learning outcomes, and (4) assessment of educational content. |
Study design | Both empirical and theoretical research published in English between 2006-2017 from peer reviewed journals, conference papers. Systematic reviews and meta-analyses will not be included. Opinion papers will be excluded. |
Literature search exclusion chart.
A specific data extraction tool was developed based on the inclusion and exclusion criteria. Each article was reviewed and information was extracted in relation to participant type, study design, sample size, types of TEL used, TEL evaluation tool used, and key study findings.
Critical appraisal is a standardized way of assessing research so that decisions can be made based on the best evidence available [
A total of 430 articles were identified in the initial titles search of the online databases and hand search of the literature. Following the application of the inclusion criteria to the titles and abstracts, 60 articles remained. The full text of the 60 articles were evaluated using the inclusion criteria and a total of 21 articles [
The results of the quality assessment are summarized within
These elements give a useful starting point in describing a requisite information set for inclusion within all good quality research studies. The value of studies that fail to include and compliment these basic elements with additional standard or sufficient research information data is significantly reduced. The studies examined had low scores for the Ethics section overall. This section looked for information related to consideration of standard research ethics, such as participant ethics and researcher ethics, even where formal ethical approval had not been required. Information that conveys ethical considerations is a prerequisite of all research studies.
The relatively low scores achieved in relation to Design, Data Collection, and Results categories are also concerning, as this renders many of the studies difficult (if not impossible) to replicate. This section looked for inclusion of information on interventions, outcomes, or treatment measures, in addition to sufficient descriptions of the research design and rationale. A key requisite of effective TEL evaluation research is ensuring that the intervention is sufficiently described to support others, who may wish to make comparisons, or to confidently apply the research to their own practice or education program development [
A subset of the studies [
Crowe Critical Appraisal Tool (CCAT) scores summary. Italics indicate studies that scored less than the average score of 68%. N/A: not applicable.
Authors | CCAT category | Raw score |
|||||||
Preliminaries | Introduction | Design | Sampling | Data |
Ethical |
Results | Discussion | ||
Akroyd et al [ |
2 | 1 | N/A | N/A | N/A | N/A | N/A | N/A | 0 |
Lotrecchiano et al [ |
1 | 1 | N/A | N/A | N/A | N/A | N/A | N/A | 0 |
Westbrook [ |
2 | 4 | 1 | 1 | 3 | 0 | 1 | 3 | 15 (37.5) |
4 | 3 | 1 | 2 | 1 | 0 | 1 | 4 | 16 (40) | |
4 | 3 | 2 | 2 | 2 | 0 | 2 | 4 | 19 (47.5) | |
2 | 4 | 2 | 2 | 2 | 1 | 3 | 4 | 20 (50) | |
3 | 4 | 3 | 3 | 4 | 0 | 2 | 3 | 21 (52.5) | |
2 | 2 | 3 | 2 | 2 | 5 | 4 | 2 | 22 (55) | |
3 | 4 | 4 | 2 | 3 | 2 | 3 | 4 | 23 (57.5) | |
4 | 5 | 3 | 4 | 1 | 2 | 2 | 4 | 25 (62.5) | |
3 | 4 | 3 | 4 | 4 | 0 | 4 | 4 | 26 (65) | |
Goldberg Goetz et al [ |
5 | 5 | 3 | 5 | 4 | 0 | 3 | 5 | 30 (75) |
Byrne et al [ |
3 | 5 | 4 | 4 | 4 | 4 | 4 | 4 | 32 (80) |
Bekkers et al [ |
5 | 5 | 4 | 5 | 2 | 3 | 4 | 4 | 32 (80) |
Popescu et al [ |
5 | 5 | 4 | 4 | 4 | 2 | 4 | 4 | 32 (80) |
Sranacharoenpong et al [ |
5 | 5 | 4 | 5 | 5 | 1 | 4 | 3 | 32 (80) |
Chang et al [ |
4 | 5 | 3 | 4 | 4 | 4 | 5 | 4 | 33 (82.5) |
Moreira et al [ |
5 | 5 | 4 | 5 | 5 | 0 | 5 | 5 | 34 (85) |
Fontaine et al [ |
5 | 5 | 3 | 5 | 5 | 2 | 5 | 5 | 35 (87.5) |
Schneiderman et al [ |
5 | 5 | 4 | 4 | 4 | 4 | 4 | 5 | 35 (87.5) |
Cortese-Peske [ |
5 | 5 | 4 | 5 | 4 | 5 | 5 | 5 | 38 (95) |
Despite the growth in popularity and types of TEL education programs produced over the last two decades, this review was only able to identify a small pool of studies that met the inclusion criteria for TEL evaluation of a health care professional education program. Many of the included studies described the TEL methods evaluated as virtual learning environments, online or e-learning modules, platforms, or blended formats. There was little evidence provided within the selected studies regarding evaluation of bidirectional TEL approaches or newer types of TEL approaches such as Personal Learning Environments (PLEs). PLEs are activity spaces in which students interact and communicate with one another, and with experts, by using Web 2.0 tools. The ultimate result of using Web 2.0 tools is the development of collective learning approaches such as “just-in-time” and “at-your-fingertips” learning opportunities that can support a wide range of teaching and learning activities [
This review of the literature concerning evaluations of TEL highlights the pattern that previous authors have noted for studies to employ a narrow focus in evaluation on either the technology equipment itself, measurement of learner satisfaction, or preknowledge and postknowledge scores [
Testing methods can heavily influence the learner’s focus and how they approach learning. If we consider that one of the key functions of employing technology in learning is to help people connect more effectively with each other and the learning materials, and to inspire learner interaction in accordance with a social constructivist learning approach, then assessments that focus solely on the work of each individual (ie, cognitivist style, pretest and posttest scoring) may have a considerable impact on each learner’s behavior and the efficacy of the program overall [
There is a need to be able to identify high quality TEL for health care education research studies and to be able to compare the outcomes from these sources to produce practical TEL evaluation tools. This need has also been highlighted throughout the last two decades in the literature on TEL in other contexts [
The CCAT tool used in this study enabled various forms of evidence presented within the literature to be explored in terms of TEL evaluation and the quality of the evidence presented. While some of the studies did present a full range of detailed research information, a number of those examined lacked information on the fundamental elements of good quality research. Ellaway [
This review has not measured standard educational quality parameters or set out to identify the requisite elements of a robust TEL evaluation guide (or tool) for health care professionals’ education. Instead, it focused on identifying what evidence of TEL evaluation for health care professional education already exists within the literature and examined whether it was of sufficient quantity and quality as an evidence base for organizations to use to develop increasingly effective and transformative TEL education programs. Although the subject of TEL dates to at least the 1990s in the context of further and higher education, its application within health care education is much more recent. On that basis, the decision was made to restrict the literature search to 2006-2017. Other databases such as Web of Science were checked but yielded few references that fully met the inclusion criteria, and such databases were therefore not included within the literature review methods for this study.
This review found limited published evidence of standard tools being implemented to measure TEL in health care education programs. Developing and implementing TEL health care education can require organizations to make considerable financial, human, and infrastructure investment. There is a mismatch between the scale of uptake of TEL in health care education and availability of a sufficiently robust evidence base of meaningful TEL evaluations in health care education. The outcomes of the systematic review and critical appraisal of this study support the views of Kirkwood and Price [
A review of the TEL evaluation literature to help identify an evidence-based list of essential parameters to include within TEL health care education evaluation reports and studies would be a useful focus for further research. There continues to be a need to develop effective and standard TEL evaluation tools and for the publication of good quality studies that describe effective evaluation of TEL education for health care professionals
There is a range of guides, frameworks, and standards emerging in the literature and across practice to guide the design of TEL within health care and higher education institutions, programs, and resources. The models that have been proposed require widespread implementation, rigorous in-practice testing, and effective reporting to ensure that TEL education programs for health care professionals are evaluated in a more robust manner than is currently evident in the literature [
Electronic database searches.
Summary of information from studies.
Crowe Critical Appraisal Tool
continuous professional development
Population, Intervention, Comparison, and Outcomes
Personal Learning Environment
technology-enhanced learning
The authors wish to acknowledge funding from National Health Service Highland and the University of the Highlands and Islands that supported this research study.
None declared.