The Karma system is currently undergoing maintenance (Monday, January 29, 2018).
The maintenance period has been extended to 8PM EST.

Karma Credits will not be available for redeeming during maintenance.
Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 11.04.18 in Vol 20, No 4 (2018): April

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/9085, first published Sep 29, 2017.

This paper is in the following e-collection/theme issue:

    Review

    Evaluation of Technology-Enhanced Learning Programs for Health Care Professionals: Systematic Review

    1National Health Service Education for Scotland, Inverness, United Kingdom

    2University of the Highlands and Islands, Inverness, United Kingdom

    3Cardiff University, Heathpark Campus, Cardiff, United Kingdom

    Corresponding Author:

    Pam Nicoll, BSc (Hon), Dip APS

    National Health Service Education for Scotland

    Old Perth Road

    Inverness, IV2 3JH

    United Kingdom

    Phone: 44 01463255 ext 7910

    Email:


    ABSTRACT

    Background: Technology-enhanced learning (TEL) programs are increasingly seen as the way in which education for health care professionals can be transformed, giving access to effective ongoing learning and training even where time or geographical barriers exist. Given the increasing emphasis on this mode of educational support for health care practitioners, it is vital that we can effectively evaluate and measure impact to ensure that TEL programs are effective and fit for purpose. This paper examines the current evidence base for the first time, in relation to the evaluation of TEL programs for health care professionals.

    Objective: We conducted a systematic review of the current literature relating to the evaluation of TEL programs for health care professionals and critically appraised the quality of the studies.

    Methods: This review employed specific search criteria to identify research studies that included evaluation of TEL for health care professionals. The databases searched included Medline Ovid, Cumulative Index of Nursing and Allied Health Literature Plus Advanced, Applied Social Sciences Index and Abstracts, ZETOC, Institute of Electrical and Electronics Engineers Explore Digital Library, Allied and Complementary Medicine, and Education Resources Information Center between January 2006 and January 2017. An additional hand search for relevant articles from reference lists was undertaken. Each of the studies identified was critically appraised for quality using the Crowe Critical Appraisal Tool. This approach produced a percentage total score for each study across specified categories. A proportion of the studies were independently assessed by an additional two reviewers.

    Results: The review identified 21 studies that met the inclusion criteria. The studies included scored totals across eight categories within a range of 37%-95% and an average score of 68%. Studies that measured TEL using learner satisfaction surveys, or combined pretest and posttest knowledge score testing with learner satisfaction surveys, were found to be the most common types of TEL evaluations evident in the literature. The studies reviewed had low scores across reporting on ethical matters, design, and data collection categories.

    Conclusions: There continues to be a need to develop effective and standard TEL evaluation tools, and good quality studies that describe effective evaluation of TEL education for health care professionals. Studies often fail to provide sufficient detail to support transferability or direct future TEL health care education programs.

    J Med Internet Res 2018;20(4):e131

    doi:10.2196/jmir.9085

    KEYWORDS



    Introduction

    The term technology-enhanced learning (TEL) is often used to describe a broad field of digital technologies used to support and mediate educational activities [1]. In this review, the term TEL is used to describe activities that are totally digitally mediated and those that are blended with more traditional educational approaches. The last two decades have seen considerable growth in the use of technology within higher education at undergraduate and postgraduate levels across the world [2]. Effective and ongoing continuous professional development (CPD) and education are essential to the delivery of high quality health and care services. TEL is increasingly presented as a means by which learners can be provided with enhanced or transformed educational experiences.

    A range of published reports have highlighted TEL as an effective method to support health care education [3,4]. In their e-learning strategy, The Higher Education Funding Council England [5] summarized three levels of potential benefits of TEL: (1) e fficiency, whereby existing processes can be carried out in a more cost-effective, time-effective, sustainable, or scalable manner; (2) enhancement, which improves existing processes and outcomes; and (3) t ransformation, representing radical change in existing processes or the introduction of new processes. The recognition of the need for continuing education and effective work-based training to support health care professionals to deliver good quality, safe, and effective care is widely accepted [6,7]. The increasing demands for effective and affordable health care education in light of resource and time constraints, together with improved access to hardware, software, and the popularity of blended learning formats, means that TEL is increasingly considered an ideal approach within health care education.

    The general availability of mobile and flexible technologies enables learners to minimize time away from health care settings to undertake training and to engage with learning resources when and where they are most suitable to their needs [8]. TEL offers a range of specific advantages for health care education, given the flexibility to update learning resources in a fast-changing field, and the scope offered for learners to share knowledge and learn critical clinical skills and decision making safely in nonclinical environments [9].

    The ability to demonstrate the added value and impact of TEL for health care education remains challenging. Previous authors have captured the nature of the challenges in the review and evaluation of TEL within medical education [10-12]. Pickering and Joynes [13] highlighted the lack of robust evidence and meaningful evaluation to support widespread implementation of TEL resources. The main challenges concern a lack of clarity around the purpose of evaluation, comprehensiveness, depth, and methodology choice to support development of the required evidence base, upon which to build effective future TEL health care programs. We need reliable, practical mechanisms to evaluate: the value for money; equity of access; and learner, service, and organizational benefits that TEL may bring [14]. There is a need to critically examine the literature on TEL implementation and evaluation within health care to guide production and implementation of effective TEL health care education programs now and in the future.

    A range of studies exist in the TEL literature which document implementation of TEL within medical and health care educational approaches. However, studies demonstrating a comprehensive TEL evaluation or use of standardized TEL evaluation tools in practice are fewer in number. Previous authors such as Ellaway [10,11], Cook and Ellaway [12], and Pickering and Joynes [13] (amongst others) have highlighted the need for robust evaluation, and have set out to develop both TEL educational standards and frameworks for evaluating TEL in medical education. Cook and Ellaway [12] have proposed a general model for evaluation. Pickering and Joynes [13] have proposed what they consider to be a more holistic TEL evaluation model for medical education.

    This systematic review of the literature aimed to identify studies that have implemented TEL evaluation for CPD and postgraduate or work-based TEL health care education programs, and to assess these using a published critical appraisal tool. The studies identified provide an evidence base for the evaluation and development of future TEL programs for health care professionals.


    Methods

    Design

    The review was carried out using a systematic integrative review method. This method allows for the inclusion of empirical and theoretical literature and quantitative and qualitative studies. This method enabled an increased number of studies to be included in the review and is appropriate for the review of evidence to highlight gaps in the literature [15].

    Eligibility and Inclusion

    The inclusion and exclusion criteria for this review were developed using the Population, Intervention, Comparison, and Outcomes (PICOS) [16] framework for systematic reviews, which is illustrated in Table 1. In devising the search strategy for this study, the PICOS framework has been used as a search tool and as an organizing framework to list terms by the main concepts in the search question. This framework is commonly used to identify components of clinical evidence for systematic reviews in evidence-based medicine and is endorsed by the Cochrane Collaboration [16].

    Search Strategy

    The following electronic databases were searched: Medline Ovid, Cumulative Index of Nursing and Allied Health Literature (CINAHL) Plus Advanced, Applied Social Sciences Index and Abstracts (ASSIA), ZETOC, Institute of Electrical and Electronics Engineers (IEEE) Explore Digital Library, Allied and Complementary Medicine (AMED), and Education Resources Information Center (ERIC) between January 2006 and January 2017 (see Multimedia Appendix 1 for details). The search was conducted using three concepts (and appropriate synonyms for each) across the selected databases: technology-enhanced learning, health care, and educational measurement. A total of 13 synonyms were used in the literature search. These phrases included technology-enhanced learning, technology-enhanced education, e-learning, e-education, blended learning, blended education, digital learning, digital education, evaluation studies, program evaluation, effectiveness, validation studies, and intervention. Reference lists were hand searched for relevant studies. The flow diagram in Figure 1 illustrates the search strategy.

    Table 1. Application of the Population, Intervention, Comparison, and Outcomes (PICOS) Framework to the research question. TEL: technology-enhanced learning.
    View this table
    Figure 1. Literature search exclusion chart.
    View this figure

    Data Extraction

    A specific data extraction tool was developed based on the inclusion and exclusion criteria. Each article was reviewed and information was extracted in relation to participant type, study design, sample size, types of TEL used, TEL evaluation tool used, and key study findings.

    Quality Assessment

    Critical appraisal is a standardized way of assessing research so that decisions can be made based on the best evidence available [17]. The Crowe Critical Appraisal Tool (CCAT) was developed to provide a tool that can reliably assess a range of research designs, provide a comprehensive appraisal approach, and provide a suitable scoring method [17]. Quality assessment was conducted for all articles included within the study. The CCAT was used to assess quality across reporting items in eight categories: Preamble, Introduction, Design, Sampling, Data collection, Ethical matters, Results, and Discussion. The items were rated on a nominal scale (Present/Absent/Not applicable). The CCAT was selected as an appropriate critical appraisal tool for this study, as it can be reliably applied across a range of study designs and has been applied to a range of both quantitative and qualitative studies [17]. The CCAT emphasizes the importance of measuring and recording scores for each of the categories rather than simply the final score for each study. This approach prevents papers that score high overall, but poorly in one or more categories, from becoming less visible than papers that score highly throughout all categories. A subset of the studies [14,18-20] was randomly selected for scoring by additional reviewers (SM and HVW).


    Results

    Search Outcome

    A total of 430 articles were identified in the initial titles search of the online databases and hand search of the literature. Following the application of the inclusion criteria to the titles and abstracts, 60 articles remained. The full text of the 60 articles were evaluated using the inclusion criteria and a total of 21 articles [14,18-37] were identified as suitable for the review.

    Multimedia Appendix 2 shows a summary of the information extracted from the studies. Ten of the studies measured TEL using learner satisfaction surveys, 8 combined pretest and posttest knowledge score testing with learner satisfaction surveys (1 with pretest and posttest knowledge score testing only), and 2 used qualitative frameworks. The selected studies described the use of a variety of different TEL modes or combinations of each, including: 12 used Web-based e-learning, 4 used learning management systems, 2 used video simulation, and 6 used blended learning formats.

    Quality Assessment

    The results of the quality assessment are summarized within Table 2. The CCAT scores for the studies selected using the inclusion criteria indicate that the quality of studies varied greatly across the range of research parameters examined. Of the 21 papers, 19 presented sufficient information to be included in the CCAT evaluation. Two papers [14,18] were identified as TEL articles rather than research papers with CCAT scores in early sections too low to be viable to continue scoring, in accordance with CCAT user guidance. Ten studies scored above the average score of 68% overall and 9 scored less than this average (these are italicized within Table 2). The lowest average-scored sections for the set of 19 studies were in Ethical Matters, with a score of 2/5. Design, Data Collection, and Results each produced average scores of 3/5. Preliminaries, Discussion, Introduction, and Sampling sections each produced the higher average scores of 4/5 for the studies examined.

    These elements give a useful starting point in describing a requisite information set for inclusion within all good quality research studies. The value of studies that fail to include and compliment these basic elements with additional standard or sufficient research information data is significantly reduced. The studies examined had low scores for the Ethics section overall. This section looked for information related to consideration of standard research ethics, such as participant ethics and researcher ethics, even where formal ethical approval had not been required. Information that conveys ethical considerations is a prerequisite of all research studies.

    The relatively low scores achieved in relation to Design, Data Collection, and Results categories are also concerning, as this renders many of the studies difficult (if not impossible) to replicate. This section looked for inclusion of information on interventions, outcomes, or treatment measures, in addition to sufficient descriptions of the research design and rationale. A key requisite of effective TEL evaluation research is ensuring that the intervention is sufficiently described to support others, who may wish to make comparisons, or to confidently apply the research to their own practice or education program development [11]. Only in this way will we begin to establish a reliable evidence base around TEL evaluation. The higher average quality assessment scores were in relation to the Discussion section and Preliminary and Introductory elements such as title, abstract, background, and objectives information.

    A subset of the studies [14,18-20] was randomly selected for additional scoring by PN. The studies were provided to two of the coauthors of this paper (SM and HVW) for them to provide an additional score using the CCAT scoring template and guide. The average difference between the original and additional scoring across 4 of the papers was found to be 19%.

    Table 2. Crowe Critical Appraisal Tool (CCAT) scores summary. Italics indicate studies that scored less than the average score of 68%. N/A: not applicable.
    View this table

    Discussion

    Principal Findings

    Despite the growth in popularity and types of TEL education programs produced over the last two decades, this review was only able to identify a small pool of studies that met the inclusion criteria for TEL evaluation of a health care professional education program. Many of the included studies described the TEL methods evaluated as virtual learning environments, online or e-learning modules, platforms, or blended formats. There was little evidence provided within the selected studies regarding evaluation of bidirectional TEL approaches or newer types of TEL approaches such as Personal Learning Environments (PLEs). PLEs are activity spaces in which students interact and communicate with one another, and with experts, by using Web 2.0 tools. The ultimate result of using Web 2.0 tools is the development of collective learning approaches such as “just-in-time” and “at-your-fingertips” learning opportunities that can support a wide range of teaching and learning activities [38]. The evaluation studies identified in this review relied heavily on measuring TEL using learner satisfaction measures. Only one study [37] cited use of a structured approach to evaluating the more interactive two-way learning process between learner and tutor that is offered by models such as the Salmon-5 stage model [39]. The most widely cited types of learner outcome measurements used within educational evaluations are Kirkpatrick’s [40] models. While these methods of measurement may often provide a useful starting point for effective assessment of TEL evaluations, the model itself may not be ideally suited to the evaluation of TEL health care education. That is, where the Kirkpatrick model emphasizes increased confidence in newly acquired knowledge as being important, effective health care education would want evidence that this new knowledge is both learned and implemented in practice beyond a practitioner’s own perception of knowledge gain or confidence [13]. For health care education to truly support health care practice we need to be able to accurately measure the added knowledge, skills, or awareness that TEL programs may or may not provide.

    This review of the literature concerning evaluations of TEL highlights the pattern that previous authors have noted for studies to employ a narrow focus in evaluation on either the technology equipment itself, measurement of learner satisfaction, or preknowledge and postknowledge scores [41,42]. Some of the studies included aspects of intralearner activity within their evaluation [18,22,27,28,30,33,37]. However, the dominance in the literature of evaluation studies measuring largely (or only) individual pretest and posttest knowledge presents a number of concerns within TEL for health care professionals. That is, where program assessments are relied upon to determine added value, learner gain, or improvement it is necessary to consider the extent to which they were matched to the actual TEL aims or enhancement being sought.

    Testing methods can heavily influence the learner’s focus and how they approach learning. If we consider that one of the key functions of employing technology in learning is to help people connect more effectively with each other and the learning materials, and to inspire learner interaction in accordance with a social constructivist learning approach, then assessments that focus solely on the work of each individual (ie, cognitivist style, pretest and posttest scoring) may have a considerable impact on each learner’s behavior and the efficacy of the program overall [43]. There is an ongoing need for more TEL evaluation studies to detail the purpose of TEL interventions and the assessment and overall approaches adopted, and to demonstrate how the technology is enhancing the learning experience [2].

    There is a need to be able to identify high quality TEL for health care education research studies and to be able to compare the outcomes from these sources to produce practical TEL evaluation tools. This need has also been highlighted throughout the last two decades in the literature on TEL in other contexts [44]. A more specific and standard approach to TEL evaluation in health care education with common measurements or tools would enable health care professionals, employers, and program designers to measure and then document effective TEL for health care education. In this way, a reliable TEL evidence base can grow and be progressively used to its full effect to influence or even transform educational programs for health care education now and in the future.

    The CCAT tool used in this study enabled various forms of evidence presented within the literature to be explored in terms of TEL evaluation and the quality of the evidence presented. While some of the studies did present a full range of detailed research information, a number of those examined lacked information on the fundamental elements of good quality research. Ellaway [11] has highlighted similar issues related to published research for online learning where, as with TEL study interventions, they are often inadequately described. A more complex analysis of interassessor consistency in scoring using the CCAT tool could have been undertaken within this study. However, although the interassessor scoring was limited in this case by available resources, it was already building on the robust methods outlined by Crowe et al [17].

    Limitations

    This review has not measured standard educational quality parameters or set out to identify the requisite elements of a robust TEL evaluation guide (or tool) for health care professionals’ education. Instead, it focused on identifying what evidence of TEL evaluation for health care professional education already exists within the literature and examined whether it was of sufficient quantity and quality as an evidence base for organizations to use to develop increasingly effective and transformative TEL education programs. Although the subject of TEL dates to at least the 1990s in the context of further and higher education, its application within health care education is much more recent. On that basis, the decision was made to restrict the literature search to 2006-2017. Other databases such as Web of Science were checked but yielded few references that fully met the inclusion criteria, and such databases were therefore not included within the literature review methods for this study.

    Conclusions

    This review found limited published evidence of standard tools being implemented to measure TEL in health care education programs. Developing and implementing TEL health care education can require organizations to make considerable financial, human, and infrastructure investment. There is a mismatch between the scale of uptake of TEL in health care education and availability of a sufficiently robust evidence base of meaningful TEL evaluations in health care education. The outcomes of the systematic review and critical appraisal of this study support the views of Kirkwood and Price [2] who stated that there is a scarcity of published studies of practical TEL education programs that generate evidence that is appropriate to the interventions described, and that can be drawn upon.

    A review of the TEL evaluation literature to help identify an evidence-based list of essential parameters to include within TEL health care education evaluation reports and studies would be a useful focus for further research. There continues to be a need to develop effective and standard TEL evaluation tools and for the publication of good quality studies that describe effective evaluation of TEL education for health care professionals . Studies often fail to provide sufficient detail to support transferability or direct future TEL health care education programs’ design and implementation. The use of a standard, practical, and quality approach to TEL evaluation, recording, and reporting with the same tools (or even parameters) across a variety of health care education programs would address this gap over time. This type of approach would also reduce duplication of efforts for organizations in creating or recreating tools, and importantly support cross-program and cross-organizational comparisons.

    There is a range of guides, frameworks, and standards emerging in the literature and across practice to guide the design of TEL within health care and higher education institutions, programs, and resources. The models that have been proposed require widespread implementation, rigorous in-practice testing, and effective reporting to ensure that TEL education programs for health care professionals are evaluated in a more robust manner than is currently evident in the literature [13]. In this way, such programs can then usefully shape the emerging field of TEL evaluation for health care education.

    Acknowledgments

    The authors wish to acknowledge funding from National Health Service Highland and the University of the Highlands and Islands that supported this research study.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Electronic database searches.

    PDF File (Adobe PDF File), 540KB

    Multimedia Appendix 2

    Summary of information from studies.

    PPTX File, 58KB

    References

    1. Goodyear P, Retalis S. Technology-Enhanced Learning: Design Patterns and Pattern Language. Rotterdam, Netherlands: Sense Publishers; 2010:1-27.
    2. Kirkwood A, Price L. Technology-enhanced learning and teaching in higher education: what is ‘enhanced’ and how do we know? A critical literature review. Learn Media Technol 2013 Feb 20;39(1):6-36. [CrossRef]
    3. Carroll C, Booth A, Papaioannou D, Sutton A, Wong R. UK health-care professionals' experience of on-line learning techniques: a systematic review of qualitative data. J Contin Educ Health Prof 2009;29(4):235-241. [CrossRef] [Medline]
    4. Childs S, Blenkinsopp E, Hall A, Walton G. Effective e-learning for health professionals and students--barriers and their solutions. A systematic review of the literature--findings from the HeXL project. Health Info Libr J 2005 Dec;22 Suppl 2:20-32 [FREE Full text] [CrossRef] [Medline]
    5. The Higher Education Funding Council for England. Digital Education Resource Archive (DERA). England: Higher Education Funding Council for England (HEFCE); 2009. Enhancing learning and teaching through the use of technology: a revised approach to HEFCE’s strategy for e-learning   URL: http://dera.ioe.ac.uk/140/ [accessed 2018-03-12] [WebCite Cache]
    6. Schostak J, Davis M, Hanson J, Schostak J, Brown T, Driscoll PS, et al. 'Effectiveness of Continuing Professional Development' project: a summary of findings. Med Teach 2010;32(7):586-592. [CrossRef] [Medline]
    7. Health Education England. 2016 Mar. Improving Safety Through Education and Training Report by the Commission on Education and Training for Patient Safety   URL: http://http/www.hee.nhs.uk/the-commission-on-education-and-training-for-patient-safety [accessed 2018-03-12] [WebCite Cache]
    8. Lupton D. Digital sociology: an introduction. SSRN Journal 2012 Aug 15:1-16 [FREE Full text] [CrossRef]
    9. Santy J, Smith L. Being an e-learner in health and social care: a student's guide. Abingdon, England: Routledge; 2007:1-33.
    10. Ellaway R. Weaving the 'e's together. Med Teach 2006 Nov;28(7):587-590. [CrossRef] [Medline]
    11. Ellaway R. Apples and architraves: a descriptive framework for e-learning research. Med Teach 2010 Jan;32(1):95-97. [CrossRef] [Medline]
    12. Cook D, Ellaway RH. Evaluating technology-enhanced learning: a comprehensive framework. Med Teach 2015;37(10):961-970. [CrossRef] [Medline]
    13. Pickering JD, Joynes VCT. A holistic model for evaluating the impact of individual technology-enhanced learning resources. Med Teach 2016 Dec;38(12):1242-1247. [CrossRef] [Medline]
    14. Akroyd M, Jordan G, Rowlands P. Interprofessional, simulation-based technology-enhanced learning to improve physical health care in psychiatry: the recognition and assessment of medical problems in psychiatric settings course. Health Informatics J 2016 Jun;22(2):312-320. [CrossRef] [Medline]
    15. Whittemore R, Knafl K. The integrative review: updated methodology. J Adv Nurs 2005 Dec;52(5):546-553. [CrossRef] [Medline]
    16. Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res 2014 Nov 21;14:579 [FREE Full text] [CrossRef] [Medline]
    17. Crowe M, Sheppard L, Campbell A. Reliability analysis for a proposed critical appraisal tool demonstrated value for diverse research designs. J Clin Epidemiol 2012 Apr;65(4):375-383. [CrossRef] [Medline]
    18. Lotrecchiano G, McDonald P, Lyons L, Long T, Zajicek-Farber M. Blended learning: strengths, challenges, and lessons learned in an interprofessional training program. Matern Child Health J 2013 Nov;17(9):1725-1734. [CrossRef] [Medline]
    19. Moreira I, Ventura SR, Ramos I, Rodrigues PP. Development and assessment of an e-learning course on breast imaging for radiographers: a stratified randomized controlled trial. J Med Internet Res 2015 Jan 05;17(1):e3 [FREE Full text] [CrossRef] [Medline]
    20. Schneiderman J, Corbridge S, Zerwic JJ. Demonstrating the effectiveness of an online, computer-based learning module for arterial blood gas analysis. Clin Nurse Spec 2009;23(3):151-155. [CrossRef] [Medline]
    21. Bekkers M, Simpson SA, Dunstan F, Hood K, Hare M, Evans J, STAR Study Team. Enhancing the quality of antibiotic prescribing in primary care: qualitative evaluation of a blended learning intervention. BMC Fam Pract 2010 May 07;11:34 [FREE Full text] [CrossRef] [Medline]
    22. Byrne E, Donaldson L, Manda-Taylor L, Brugha R, Matthews A, MacDonald S, et al. The use of technology enhanced learning in health research capacity development: lessons from a cross country research partnership. Global Health 2016 May 10;12(1):19 [FREE Full text] [CrossRef] [Medline]
    23. Chang W, Hsiao SS, Chang PC, Lee PH. Developing an E-learning education programme for staff nurses: processes and outcomes. Nurse Educ Today 2008 Oct;28(7):822-828. [CrossRef] [Medline]
    24. Chuo Y, Liu C, Tsai C. Effectiveness of e-learning in hospitals. Technol Health Care 2015;23 Suppl 1:S157-S160. [CrossRef] [Medline]
    25. Cortese-Peske M. Icahn School of Medicine at Mount Sinai.: ERIC; 2013. Improving cultural competency and disease awareness among oncology nurses caring for adult T-cell leukemia and lymphoma patients   URL: http://www.proquest.com/en-US/products/dissertations/individuals.shtml [accessed 2017-03-09] [WebCite Cache]
    26. Fontaine G, Cossette S, Heppell S, Boyer L, Mailhot T, Simard MJ, et al. Evaluation of a Web-based E-learning platform for brief motivational interviewing by nurses in cardiovascular care: a pilot study. J Med Internet Res 2016 Aug 18;18(8):e224 [FREE Full text] [CrossRef] [Medline]
    27. Gill A. E-learning and professional development--never too old to learn. Br J Nurs 2007;16(17):1084-1088. [CrossRef] [Medline]
    28. Goldberg GD, Clement DG, Cotter JJ. Development and alumni assessment of an interdisciplinary PhD program offered through a blended learning environment. J Allied Health 2011;40(3):137-142. [Medline]
    29. Heartfield M, Morello A, Harris M, Lawn S, Pols V, Stapleton C, et al. E-learning competency for practice nurses: an evaluation report. Aust J Prim Health 2013;19(4):287-291. [CrossRef] [Medline]
    30. Ingelbeen B, Aryeetey R, Khanna J, Mahamoudane N, Vincent D, Kiyan C, et al. Exploring three approaches to offer distance learning courses through a social network of health researchers in three African countries. In: IEEE Xplore.: IEE Explore; 2014 May 9 Presented at: International School of Tanganyika-Africa Conference Proceedings, 2014; 2014; Le Meridien Ile Maurice, Mauritius   URL: http/ieeexplore.ieee.org [CrossRef]
    31. Konstantinidis S, Kitsou A, Bamidis P, Mikos T, Zafrakas M, Nikolaidou M, et al. A SCORM compliant e-learning platform for cervical cancer pPrevention. IEEE Xplore; 2008 Jun 19 Presented at: Computer-Based Medical Systems, 2008; June 19, 2008; Jyvaskyla, Finland. [CrossRef]
    32. Popescu D, Avramescu ET, Pascual LJ, Pratikakis I, Perantonis S, Lupescu O. A Web-based e-training platform for biomedical engineering education. 2014 Jun 19 Presented at: 2014 International Conference and Exposition on Electrical and Power Engineering; 2014; Iasi, Romania   URL: http://ieeexplore.ieee.org/document/6969889/ [CrossRef]
    33. Safwat A, Pourabdollah A. E-learning for healthcare professionals towards HIS in Egypt. 2009 Dec 16 Presented at: 7th International Conference on Communications and Information Technology (ICICT); December 16, 2009; Cairo, Egypt   URL: http://ieeexplore.ieee.org
    34. Sranacharoenpong K, Hanning RM, Sirichakwal PP, Chittchang U. Process and outcome evaluation of a diabetes prevention education program for community healthcare workers in Thailand. Educ Health (Abingdon) 2009 Dec;22(3):335. [Medline]
    35. Walsh K, Rafiq I, Hall R. Online educational tools developed by Heart improve the knowledge and skills of hospital doctors in cardiology. Postgrad Med J 2007 Jul;83(981):502-503 [FREE Full text] [CrossRef] [Medline]
    36. Wang F. Valuation of Online Continuing Medical Education and Telemedicine in Taiwan. J Educ Techno Soc 2008 Jan 01;11(4):190 [FREE Full text]
    37. Westbrook C. Online collaborative learning in health care education. EURODL 2012 Jan 26;1:1-6 [FREE Full text]
    38. Rahimi E, van den Berg J, Veen W. A learning model for enhancing the student's control in educational process using Web 2.0 personal learning environments. Br J Educ Technol 2014 May 13;46(4):780-792. [CrossRef]
    39. Salmon G. E-moderating: the kiey to teaching and learning online. London: Routledge; Jan 22, 2004.
    40. Kirkpatrick Partners. 2018. The new world Kirkpatrick model   URL: https://www.kirkpatrickpartners.com/ [accessed 2016-05-12] [WebCite Cache]
    41. Koehler N. Medical students’ use of and attitudes towards medical applications. Journal MTM 2012 Dec 22;1(4):16-21. [CrossRef]
    42. Wallace S, Clark M, White J. 'It's on my iPhone': attitudes to the use of mobile computing devices in medical education, a mixed-methods study. BMJ Open 2012;2(4):2012-00109 [FREE Full text] [CrossRef] [Medline]
    43. Cubric M. Wiki-based process framework for blended learning. In: Proceeding of International Symposium on Wikis, 2007. 2007 Oct 25 Presented at: International Symposium on Wikis; October 21-25, 2007; Montreal, Canada p. 11-24. [CrossRef]
    44. Roblyer M. Educational technology research that makes a differenceries introduction. CITE 2005(Online serial, 5(2)):746-748 [FREE Full text]


    Abbreviations

    CCAT: Crowe Critical Appraisal Tool
    CPD: continuous professional development
    PICOS: Population, Intervention, Comparison, and Outcomes
    PLE: Personal Learning Environment
    TEL: technology-enhanced learning


    Edited by G Eysenbach; submitted 29.09.17; peer-reviewed by S Manca, KC Wong, R Pankomera; comments to author 26.10.17; revised version received 18.12.17; accepted 21.12.17; published 11.04.18

    ©Pam Nicoll, Sandra MacRury, Hugo C van Woerden, Keith Smyth. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 11.04.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.