Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, March 11, 2019 at 4:00 PM to 4:30 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 21.07.15 in Vol 17, No 7 (2015): July

This paper is in the following e-collection/theme issue:

    Original Paper

    A Cost-Effectiveness Analysis of Blended Versus Face-to-Face Delivery of Evidence-Based Medicine to Medical Students

    1Monash University, Frankston, Australia

    2Monash University, Caulfield, Australia

    3Centre for Health & Social Care Research, Kingston University & St George’s, University of London, London, United Kingdom

    4BMJ Learning, London, United Kingdom

    5Monash University, Melbourne, Australia

    Corresponding Author:

    Stephen Maloney, PhD

    Monash University

    PO Box 527

    Frankston,

    Australia

    Phone: 61 3 990 44240

    Fax:61 3 990 44812

    Email: stephen.maloney@monash.edu


    ABSTRACT

    Background: Blended learning describes a combination of teaching methods, often utilizing digital technologies. Research suggests that learner outcomes can be improved through some blended learning formats. However, the cost-effectiveness of delivering blended learning is unclear.

    Objective: This study aimed to determine the cost-effectiveness of a face-to-face learning and blended learning approach for evidence-based medicine training within a medical program.

    Methods: The economic evaluation was conducted as part of a randomized controlled trial (RCT) comparing the evidence-based medicine (EBM) competency of medical students who participated in two different modes of education delivery. In the traditional face-to-face method, students received ten 2-hour classes. In the blended learning approach, students received the same total face-to-face hours but with different activities and additional online and mobile learning. Online activities utilized YouTube and a library guide indexing electronic databases, guides, and books. Mobile learning involved self-directed interactions with patients in their regular clinical placements. The attribution and differentiation of costs between the interventions within the RCT was measured in conjunction with measured outcomes of effectiveness. An incremental cost-effectiveness ratio was calculated comparing the ongoing operation costs of each method with the level of EBM proficiency achieved. Present value analysis was used to calculate the break-even point considering the transition cost and the difference in ongoing operation cost.

    Results: The incremental cost-effectiveness ratio indicated that it costs 24% less to educate a student to the same level of EBM competency via the blended learning approach used in the study, when excluding transition costs. The sunk cost of approximately AUD $40,000 to transition to the blended model exceeds any savings from using the approach within the first year of its implementation; however, a break-even point is achieved within its third iteration and relative savings in the subsequent years. The sensitivity analysis indicates that approaches with higher transition costs, or staffing requirements over that of a traditional method, are likely to result in negative value propositions.

    Conclusions: Under the study conditions, a blended learning approach was more cost-effective to operate and resulted in improved value for the institution after the third year iteration, when compared to the traditional face-to-face model. The wider applicability of the findings are dependent on the type of blended learning utilized, staffing expertise, and educational context.

    J Med Internet Res 2015;17(7):e182

    doi:10.2196/jmir.4346

    KEYWORDS



    Introduction

    Evidence-based medicine (EBM) combines the best available evidence with clinical expertise and patient values [1] and is core to many medical programs worldwide [2-5]. There is an imperative to increase the number of competent EBM trained practitioners. Medical practitioners who are competent and confident in applying EBM possess a powerful tool to inform their decision making. EBM competencies also provide the ability to facilitate life-long learning, as clinicians are able to ask effective clinical questions, acquire information through emerging research, appraise its quality and relevance, apply evidence to practice, and assess its impact [1].

    Teaching and learning is undergoing a cultural change with the expansion of student-centered learning, including the push for flipped teaching, peer-assisted learning, increased use of virtual learning environments, and the use of simulation. Research has shown that Web-based learning results in improved outcomes when applied to health professional education [6], with studies focusing on clinical disciplines within medicine reporting an increase in student self-efficacy, knowledge, and self-directed learning [4,7-9]. However, due to the variety of online learning applications and education provider contexts, there is an added need for detailed and robust studies [10], as uninformed transition of learning material to an online environment can have a negative impact on educational outcomes [11]. Online education, or its many variations (eg, eLearning, Web 2.0), appears to hold great promise for addressing the accessibility and efficiency of education, yet currently there is a lack of evidence to inform educators and learners as to the most effective methods of teaching EBM to medical students.

    An investigation into the effectiveness of implementing a blended learning (BL) versus a traditional face-to-face (F2F) learning approach of teaching EBM to medical students was conducted by Ilic et al [12]. This multicenter international study used validated outcome measures of EBM competency to determine that BL is no more effective than F2F at increasing medical students’ knowledge and skills in EBM. These authors concluded that the BL approach was significantly more effective at increasing student attitudes toward EBM and self-reported use of EBM in clinical practice. Although their study looked at competency, attitudes, skills, and behavior, the missing piece of the puzzle, as acknowledged by the authors, was a measure of cost-effectiveness to help facilitate sustainable adoption of the approach.

    The cost and value of teaching and learning practices in medical education have a direct impact on the accessibility of education, the efficiency and quality of education, and the productivity of our health workforce [13-15]. Cost-effectiveness analysis allows decision makers to decrease the risks of implementation from both a financial perspective and the perspective of maintaining quality of education, thereby facilitating adoption [16]. Considerations of cost and value are most commonly known for their application to pharmaceutical interventions: impact of drug A versus drug B, with consideration of the drug’s effectiveness, side effects, and value for money. Pharmacists can still stock the most expensive choices, and consumers can still purchase them; however, these are informed decisions based on the evidence. The same principles should apply to education. For an educator to effectively review their practices and pedagogy, they must consider the learning experience and learning outcomes alongside measures of cost and value [17].

    Current literature on the cost and value of technological innovations in education is divided, with some studies supporting its cost-effectiveness [18], and others indicating that costs are higher as a result of increased resource development time and need for technological support [19,20]. Economic analysis for health education has primarily focused on telemedicine technology, medical reviews by remote physicians [21], or has been concerned with the cost-effectiveness of modalities for patient education [22]. Previous cost studies on BL approaches in health professional education have typically been conducted on short courses with small sample sizes [18,19] or were unable to determine the cost-effectiveness relationship [23,24]. Thus, there is considerable doubt as to whether existing literature could be generalized to the core-teaching content of a contemporary medical program.

    This paper presents the findings from a study that aimed to compare the cost-effectiveness of a face-to-face approach and blended learning approach for EBM training within a medical program—training that has rigorously evaluated effectiveness as tested within a randomized controlled trial (RCT).


    Methods

    Design

    The economic evaluation was conducted as part of an RCT comparing the EBM competency of medical students who participated in two different modes of education delivery [12]. The attribution and differentiation of costs between the interventions within the RCT, in conjunction with measured outcomes of effectiveness, enabled a cost-effectiveness analysis to be applied.

    Trial Participants, Methods, and Results

    The fundamental elements of the RCT have been reported to provide context for the application of the cost-effectiveness analysis. Further detail on the methods and results of the RCT are available within a published paper [12] and pilot study [4].

    A multi-campus study was performed with medical students enrolled in the Bachelor of Medicine, Bachelor of Surgery course at Monash University, Melbourne and Malaysia, including both undergraduate and post-graduate students. Participants were third-year medical students, who were all entering their first year of clinically based training and first year of formal EBM training. The EBM unit is integrated within the medical curriculum and is equivalent to a 6-credit point unit. A total of 497 students were randomized to receive EBM teaching via either the incumbent face-to-face approach (F2F) or the blended learning approach (BL). Students randomized to the intervention group received the same theoretical concepts taught in the control group, but in a BL approach. The BL approach to teaching EBM integrated (1) classroom activities (lecture/tutorial) with (2) online and (3) mobile learning as described in Table 1.

    Table 1. Detailed description of the DL and BL teaching approach to EBM with associated costs.
    View this table

    A total of 147 (29.6%) of the 497 students completed the follow-up assessments on EBM competency and attitudes. EBM competencies were assessed using the validated Berlin questionnaire [27]. Students’ self-efficacy, attitudes, and behavior were also assessed. EBM competency did not differ significantly between students receiving the BL approach versus those receiving the F2F approach: mean difference -0.68, 95% CI -1.71 to 0.34, P=.19. Although student ratings of self-efficacy, attitudes, and behaviors all displayed a significant preference for the BL model. In total, 74 students completed the F2F model of training, with a mean score of 7.98 (SD 3.35), and 73 students completed the training via the BL approach, with a mean score of 8.67 (SD 2.96).

    Economic Analysis Procedure

    The following analysis was applied from the perspective of Monash University, measuring the cost of training student clinicians against their self-reported level of EBM competence. The primary outcome was student competency in EBM, measured 1 month after the teaching activities, using the validated Berlin Questionnaire. We calculated the cost-effectiveness for each course delivery method by first determining the quality of students’ education with each method, known as quality-adjusted students educated (QASE), using the formula QASE = number of students educated x the group’s average rating on the Berlin Questionnaire. In this approach, the reported average rating was used as a surrogate for measuring the improved ability of the total cohort of 497 students for each teaching approach. QASE is the measurement of effect in the incremental cost-effectiveness analysis. Cost-effectiveness was calculated using the incremental cost-effectiveness ratio (ICER), which measures cost per QASE (Figure 1). The ICER is reflective of the ongoing operational costs and does not include the initial transition cost in its calculation. Thus, the results represent the cost-effectiveness of the second iteration and onwards only.

    Figure 1. Equation for the calculation of the incremental cost for each quality-adjusted student educated (ICER).
    View this figure

    Two rounds of ICER calculations were completed. First, an ICER comparing F2F to BL was calculated, establishing the hierarchy of cost-effectiveness. There were four possible outcomes of this analysis: (1) BL is more costly and more effective than F2F, (2) BL is more costly and less effective than F2F, (3) BL is less costly and more effective than F2F, and (4) BL is less costly and less effective than F2F [28].

    The World Health Organization recommends that cost-effectiveness comparisons should be carried out against a common baseline as this is more comparable across populations and studies [29]. Thus, in the second round of ICER calculations, F2F and BL were independently compared to no EBM training. As there are no pre-test Berlin scores from the RCT, baseline scores from third-year medical students with no prior EBM training (mean 4.2, SD 2.2) from the Berlin Questionnaire validation study by Fritsche et al were used [27]. Independent ICER calculation of F2F and BL compared to baseline calculates the cost per student per increase in QASE. Due to commercial sensitivities, these values were compared and the percentage difference reported.

    The sunk cost of transitioning to a BL format was not included within the ICER, as is typical within economic analyses. However, the transition costs are reported separately, due to its importance to decision makers considering implementation or adoption of similar pedagogy. A further present value (PV) break-even analysis incorporating transition costs was calculated using a real discount rate of 4% (Figure 2). Break-even analysis calculates the point in time at which the total running cost of the F2F approach equals the total running cost of the BL approach plus the cost of transitioning to BL. PV accounts for the time preference of money, allowing for present day comparisons to be made on future cash flows. Subsequent iterations of the program are assumed to occur at 1-year intervals. Due to commercial sensitivities, PV values are expressed as the difference between teaching methods.

    Figure 2. Equation for the PV break-even calculation where C=cost of teaching method, r=discount rate, t=number of years, and BL0=cost of transitioning to BL.
    View this figure

    Measurement of Cost

    The BL arm of the RCT has continued as the preferred method of delivery of EBM with the medical program. Costs were modeled on the 2013 RCT, and the assumptions relating to the ongoing costs were validated within the 2014 iteration of the program. The course iterations and outputs as they relate to the analysis are shown in Figure 3.

    Cost categories considered within the analysis included (1) transition costs, (2) ongoing staffing costs, (3) space charges, and (4) overheads (Table 2). All staffing hours were calculated from actual values observed in the original RCT and differentiated by academic level. Other values where modeled have been explicitly stated in Table 2, along with the assumptions made. Where appropriate, results have been expressed as the difference between F2F and BL due to commercial sensitivities.

    Table 2. Description of cost categories.
    View this table
    Figure 3. Program iterations and outputs relevant to past and current research.
    View this figure

    Sensitivity Analysis

    We included a multivariate sensitivity analysis around permutations to the key variables. Guided by available literature, scenarios were constructed around increased transition costs and increased staffing requirements for running the BL format. Thus, the robustness of our economic model was tested, allowing the reader to tailor the findings to different educational settings.


    Results

    BL and F2F Approach Delivery Inputs

    The staffing profile, wage, and difference in hours attributed to the delivery of each method are detailed in Table 3 below. Where there is a difference in hours, it is expressed as F2F hours minus BL hours; that is, positive values indicate fewer BL hours.

    Space charges for the F2F and BL teaching methods were identical. Both methods had booked rooms seating 180 people (AUD $131.34/hr) for 28 hours and rooms seating 50 people (AUD $76.61/hr) for 210 hours. Major updating of the curriculum content and materials was considered required at 5 years and estimated at 160 hours of staff time for both F2F and BL approaches.

    The cost of transitioning from the F2F to BL approach is the result of staffing time plus on-costs, valued at AUD $38,186. The difference in delivery costs is AUD $12,514 in favor of the BL method. The lower BL delivery cost is the result of lower preparation and direct teaching costs compared to the F2F method.

    Table 3. Staffing profile with difference in hours for F2F and BL approaches.
    View this table

    Cost Effectiveness

    The BL method was less costly and more effective to operate than the F2F approach. The ICER result comparing F2F to BL was -$1.10, indicating that to operate the BL model, there is a saving of $1.10 per student per increase in QASE above the QASE of the F2F method. Independent ICER calculations of F2F and BL compared to baseline found that BL was 24% more cost-effective to operate than F2F. That is, the cost of achieving a statistically similar Berlin score for the same number of students is 24% less in the BL method, excluding transition cost. Given that the BL approach is slightly more effective than the F2F approach, the BL approach would have to cost 18% more than the F2F approach to run for there to be no difference in ICER between the two approaches.

    Break-Even Analysis

    The PV difference in cost between F2F and BL approaches, including the transition costs to the BL model, are shown in Figure 4. The graphed line demonstrates the sunk cost of the transition to the blended learning approach, the cost of which is recovered by the lower running cost of the BL method in the third year. Subsequent years show a relative saving using the BL method, with PV savings of approximately AUD $17,000 after 5 years and AUD $63,000 after 10 years.

    Figure 4. The accumulated difference in the PV of cost between F2F and BL across 10 years.
    View this figure

    Sensitivity Analysis

    A multivariate sensitivity analysis was conducted with scenario variations to ongoing staffing costs and mean Berlin score (Table 4). Results are reported as the percentage difference in the independent F2F and BL ICER values. Two scenarios of educational effectiveness with the BL approach were simulated, that of mean Berlin scores as per the RCT and another with scores equivalent to the F2F method. The results show the BL format is more cost-effective to operate up to staffing levels 20% higher than F2F, using the RCT mean Berlin score.

    Table 4. Multivariate sensitivity analysis adjusting staffing levels and mean BL Berlin score.
    View this table

    Table 5 explores various transition cost scenarios and the resulting transition cost. Using the scenarios from Table 4, the analysis shows changes in 5-year accumulated PV as a result of modifying ongoing and transition costs. Negative values indicate that, at 5 years, the savings made from running the lower cost BL format have not yet overcome the transition cost. In fact, in the scenarios in the last 3 columns of the table, the cost of running the BL program is greater than or equal to that of the F2F program, and as such will never recover their transition costs.

    Table 5. Multivariate sensitivity analysis adjusting transition costs and ongoing cost scenarios.
    View this table

    Discussion

    Principal Findings

    This research aimed to determine the cost-effectiveness of an F2F and BL approach for EBM training within a medical program. The training had known outcomes evaluated through a sufficiently powered randomized controlled trial. The ICER results indicate that the BL approach provided an improved cost-effectiveness proposition from the perspective of the educational institution—costing less to train student clinicians to an equivalent level of competency. Cost-effectiveness results should be interpreted together with the break-even point of 3 years, as it is after this point that savings are realized by the institution. In addition to this, the findings by Ilic et al [12] indicated that the blended learning approach had the added benefit of increasing student attitudes toward EBM and self-reported use of EBM in clinical practice.

    Given that blended learning can take a wide variety of formats, the findings of this study show that low-cost resources such as YouTube and student self-directed activities are cost-effective at improving EBM. This may be in part due to the nature of EBM teaching, as opposed to other aspects of medical education that may tend to utilize more costly mediums such as animations or virtual patients, which few medical schools can afford to create [31]. In fact, many costly features such as animations, high-quality video products, and excessive multimedia have little added value and may actually impede learning [20]. Additionally, given that the online learning utilized either currently owned or free access resources, there were no new costs attributed to software or licensing. Prior to using the BL approach in EBM, lecture material and online resources were already available through the online learning management system. In the RCT, the combination of using simple BL formats and staff having prior experience with learning technology may have helped to exclude transition costs such as IT support, consultation, and piloting. Despite this, the impact of support roles in transitioning learning format appears to be highly situation dependent, given that a study on developing an e-module found that approximately one third of development time was attributed to administrative or technical staff [19]. The impact of IT support costs on our cost model was tested in the sensitivity analysis, calculated as an additional 210 hours of staff time to transition. Despite significantly increasing the cost to transition, over 5 years, this cost was recovered by the savings in operating costs.

    The differences in operating cost between the two approaches is dependent on staffing time, specifically the preparation and teaching time. The lower values in the BL approach can be attributed to greater emphasis on peer-to-peer learning and self-directed activities. Staff members act as facilitators rather than tutors, guiding students to resources rather than direct teaching. In contrast, previous studies have highlighted the significant increase in staffing time associated with online education methods [19,32]. Increases appear to be the result of using email, discussion forums, and online chat sessions, 75% of which occur outside regular working hours [19]. Thus, it is likely that not all BL formats would be cost-effective, and educators should carefully design their learning models around staff time, as this is the most significant cost driver. This is reflected in the sensitivity analysis showing that large increases in staffing hours above the traditional approach result in unfavorable ICER values and an inability to recover the transition cost. As would be expected, the change in teaching method had no impact on unit coordination, marking, or examination costs.

    The BL format used in this study did not reduce the face-to-face classroom hours. Rather, there was an emphasis on how existing resources were used, how time was spent outside of class, and changing what activities were completed in class. That is, the cost reduction to achieve the same or better outcome observed in this study, is naturally a reflection of productivity gains around resources used based on enhanced teaching formats. As a result, the cost-effectiveness found in this approach may not be generalizable to institutions seeking to reduce student contact hours or promote distance education. Additionally, this study benefited from economies of scale as the EBM program educates approximately 500 students across multiple campuses. Small increases in education effectiveness are magnified when the effect is across a large number of students, resulting in favorable ICER and break-even results. This multiplication effect also applies to staff time, with a YouTube video produced by one person replacing the work of many people. If preparation time is reduced by just 20 minutes for a session, across 5 sites and 10 sessions the reduction in preparation adds up to over 15 hours. Thus, when designing a cost-effective program, it appears pertinent to consider not only what aspect of learning is being changed, but how many people the change will influence.

    There may be concerns over the acceptability of a BL approach from a student or industry perspective. Social demands are driving information and communication technologies to be more commonplace and publically acceptable. It is conceivable that although the BL approach was accepted by students within this study, that this may have been a function of time, and the same approach may not have been acceptable in the preceding years. The concept of pre-class homework is not new, and changing the format does not necessarily solve the inherent issue of student adherence [20]. It remains unknown how different formats of BL will impact participation and outcomes [20]. Regardless, from the perspective of the student, pre-class activities require additional time and may be more or less efficient than traditional approaches [20]. To consider the student perspective on costs, a cost-benefit analysis using “willingness to pay” may be used. Such an analysis would help to consider student perceived value and inform changes in fee structure associated with changes in learning format.

    Limitations

    The generalizability of this study to other populations, as a result of the specific approach to BL used, have already been discussed. However, it is also necessary to discuss the limitations of the methodology used, which may influence the strength of the findings. The first limitation is the use of the ICER as a measure of cost-effectiveness. As mentioned, the standard use of the ICER does not include the investment cost of transitioning learning format. It is possible to include the investment cost into the ICER calculation for the first iteration only, as this is the period when the cash flow occurs, when considering the investment as a prospective cost. However, the ICER would not be representative of other periods. By excluding the transition cost, the ICER presented in this study reflects the ongoing operational cost-effectiveness from the second iteration onwards and must be interpreted together with the break-even analysis. Thus, programs with high set-up costs and low running costs, which may be typical of many blended learning formats, are shown to be more cost-effective by the ICER. However, the blended format in the RCT had relatively low set-up costs, which have been roughly estimated by the course developers to be similar to the cost of transitioning from a blended approach to a traditional approach. For the same reasons, the BL 5-year major curriculum update was estimated to be the same as F2F. It is logical to infer that formats with high set-up costs will incur high updating costs; however, within an educational institution, updating is often limited by time availability. Another consideration with the ICER is that it assumes a linear increase in EBM expertise. That is, the assumption that an increase in Berlin score from 30% to 40% represents the same increase in EBM expertise as an increase from 70% to 80%. There is insufficient literature on the Berlin Questionnaire to draw conclusions on the proficiency distribution, and while this does not preclude the use of the ICER calculation, it should be considered when interpreting the results.

    In the RCT itself, the quality of the teacher within both methods may also influence the costs and the educational outcomes. Within the study, this potential variable was controlled for by using the same teaching staff across both modes of education. However, in using the same staff, there is the risk that staff may inadvertently bias the results in favor of the teaching model that they are more skilled or enthusiastic about [33]. The scope of research does not include variations in the savings made by the institution. If the blended and face-to-face approaches were offered with different fee structures, the total value proposition from the educational institution will change. Less tangible benefits from the modes of delivery, such as feelings of connectedness often associated with face-to-face learning [34], flexibility [20], and other measures of the full educational experience have not been calculated, and remain as unknown for both of the educational approaches utilized [35-37]. And finally, there is the assumption that increasing the number of EBM trained practitioners, competent in incorporating best evidence, with patient values and clinical practice, actually makes a difference to patient outcomes.

    No approach to cost analysis in medical education is perfect; however, being transparent about the approach used is an important step forward considering the current state of the literature. Here, we feel we have been as transparent as possible about the costs of the components that made up the various approaches to medical education described. The most common errors in cost analyses include omitting hidden costs and paying insufficient attention to the main costs. In this regard, we feel we have gone to considerable lengths to seek out and account for hidden costs and that we have paid most attention to the main costs—that is, the costs of faculty and resources.

    Conclusions

    Under the study settings, a blended learning approach to training practitioners to be competent in applying evidenced-based medicine was more cost-effective to operate than the traditional face-to-face model. Furthermore, the BL approach resulted in significantly greater increases in student attitudes toward EBM and self-reported use of EBM in clinical practice. When taking into account the cost of transitioning to the new format, the benefit of the cost-effectiveness is realized by the institution only after the third operational year. The primary drivers of cost-effectiveness were the low-cost online resources chosen, decreased staffing levels, and economies of scale. Implementing BL is not without its risks though and requires a significant investment cost in tailoring the teaching and learning resources to the Web-based environment during the transition to this approach. Using a BL approach will not necessarily be cost-effective, and consideration should be given to the blend utilized, staff expertise, and the educational setting. Health professions’ education and educational research has developed into a respectable scientific discipline due to the shift toward scientific rigor and peer-review [38,39]. To maintain its relevance and accountability and to improve the adoption of new educational approaches and innovations, the next cultural shift in this field needs to be toward fiscal responsibility alongside learning outcomes, such as measuring outcomes of cost-effectiveness alongside measures of educational outcomes and the learning experience. The purpose of this shift is not to cut costs or to increase spending but simply to improve value.

    Authors' Contributions

    All authors have significantly contributed to the research within the manuscript, its analysis, and the writing and critical review of the final manuscript.

    Conflicts of Interest

    None declared.

    References

    1. Straus S, Glasziou P, Richardson W, Haynes B. Evidence-based medicine: how to practice and teach it. Edinburgh: Churchill Livingstone: Elsevier; 2011.
    2. Finkel ML, Brown H, Gerber LM, Supino PG. Teaching evidence-based medicine to medical students. Med Teach 2003 Mar;25(2):202-204. [CrossRef] [Medline]
    3. Meats E, Heneghan C, Crilly M, Glasziou P. Evidence-based medicine teaching in UK medical schools. Med Teach 2009 Apr;31(4):332-337. [CrossRef] [Medline]
    4. Ilic D, Hart W, Fiddes P, Misso M, Villanueva E. Adopting a blended learning approach to teaching evidence based medicine: a mixed methods study. BMC Med Educ 2013;13:169 [FREE Full text] [CrossRef] [Medline]
    5. Nair B, Coughlan J, Hensley M. Student and patient perspectives on bedside teaching. Med Educ 1997;31(5):341-346. [CrossRef]
    6. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Instructional design variations in internet-based learning for health professions education: a systematic review and meta-analysis. Acad Med 2010 May;85(5):909-922. [CrossRef] [Medline]
    7. Grasl MC, Pokieser P, Gleiss A, Brandstaetter J, Sigmund T, Erovic BM, et al. A new blended learning concept for medical students in otolaryngology. Arch Otolaryngol Head Neck Surg 2012 Apr;138(4):358-366. [CrossRef] [Medline]
    8. Lehmann R, Bosse H, Simon A, Nikendei C, Huwendiek S. An innovative blended learning approach using virtual patients as preparation for skills laboratory training: perceptions of students and tutors. BMC Medical Education 2013;13:23. [CrossRef]
    9. Woltering V, Herrler A, Spitzer K, Spreckelsen C. Blended learning positively affects students' satisfaction and the role of the tutor in the problem-based learning process: results of a mixed-method evaluation. Adv Health Sci Educ Theory Pract 2009 Dec;14(5):725-738. [CrossRef] [Medline]
    10. Ellaway R. E-learning: is the revolution over? Med Teach 2011;33(4):297-302. [CrossRef] [Medline]
    11. Maloney S, Tai J, Paynter S, Lo K, Ilic D. Self-Directed Online Learning Modules: Students’ Behaviours and Experiences. Pharmacy 2013 Jul 29;1(1):8-15. [CrossRef]
    12. Ilic D, Nordin R, Glasziou P, Tilson J, Villanueva E. A randomised controlled trial of a blended learning education intervention for teaching evidence-based medicine. BMC Medical Education 2015;15:39. [CrossRef]
    13. Kremer M, Brannen C, Glennerster R. The challenge of education and learning in the developing world. Science 2013 Apr 19;340(6130):297-300. [CrossRef] [Medline]
    14. Kremer M, Holla A. Improving Education in the Developing World: What Have We Learned from Randomized Evaluations? Annu Rev Econom 2009;1:513-542 [FREE Full text] [CrossRef] [Medline]
    15. Zurn P, Dal Poz Mario R, Stilwell B, Adams O. Imbalance in the health workforce. Hum Resour Health 2004 Dec 17;2(1):13 [FREE Full text] [CrossRef] [Medline]
    16. Walsh K. In: Walsh K, editor. Cost Effectiveness in Medical Education. United Kingdom: Radcliffe Medical PR; 2010.
    17. Walsh K, Reeves S, Maloney S. Exploring issues of cost and value in professional and interprofessional education. J Interprof Care 2014 Nov;28(6):493-494. [CrossRef] [Medline]
    18. Maloney S, Haas R, Keating JL, Molloy E, Jolly B, Sims J, et al. Breakeven, cost benefit, cost effectiveness, and willingness to pay for web-based versus face-to-face education delivery for health professionals. J Med Internet Res 2012;14(2):e47 [FREE Full text] [CrossRef] [Medline]
    19. Delgaty L. A critical examination of the time and workload involved in the design and delivery of an e-module in postgraduate clinical education. Med Teach 2013 May;35(5):e1173-e1180. [CrossRef] [Medline]
    20. Cook DA. The value of online learning and MRI: finding a niche for expensive technologies. Med Teach 2014 Nov;36(11):965-972. [CrossRef] [Medline]
    21. Whitten P, Mair F, Haycox A, May C, Williams T, Hellmich S. Systematic review of cost effectiveness studies of telemedicine interventions. BMJ 2002 Jun 15;324(7351):1434-1437 [FREE Full text] [Medline]
    22. Upton D, Upton P. Development of an evidence-based practice questionnaire for nurses. J Adv Nurs 2006 Feb;53(4):454-458. [CrossRef] [Medline]
    23. Huang G, Reynolds R, Candler C. Virtual patient simulation at US and Canadian medical schools. Acad Med 2007 May;82(5):446-451. [CrossRef] [Medline]
    24. Stotzer RL, Fujikawa K, Sur J, Arnsberger P. Cost Analysis of a Distance Education MSW Program. Journal of Teaching in Social Work 2013 Nov;33(4-5):357-368. [CrossRef]
    25. Ilic D. Year 3-EBCP. 2013.   URL: https://www.youtube.com/user/EBMOLT [accessed 2015-05-26] [WebCite Cache]
    26. Monash University. 2013. Evidence Based Practice   URL: http://guides.lib.monash.edu/medicine [accessed 2015-05-26] [WebCite Cache]
    27. Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer H, Kunz R. Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ 2002 Dec 7;325(7376):1338-1341 [FREE Full text] [Medline]
    28. Philips C, Thompson G. What is cost-effectiveness?. 2003.   URL: http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/Cost-effect.pdf [accessed 2015-05-26] [WebCite Cache]
    29. World Health Organization. Making choices in health: WHO guide to cost-effectiveness analysis. Geneva: World Health Organization; 2003.   URL: http://www.who.int/choice/publications/p_2003_generalised_cea.pdf [accessed 2015-07-16] [WebCite Cache]
    30. Monash University. 2014. Monash University Enterprise Agreements   URL: http://adm.monash.edu.au/enterprise-agreements/ [accessed 2015-05-26] [WebCite Cache]
    31. Huang G, Reynolds R, Candler C. Virtual patient simulation at US and Canadian medical schools. Acad Med 2007 May;82(5):446-451. [CrossRef] [Medline]
    32. Spector JM. Time demands in online instruction. Distance Education 2005 Jan;26(1):5-27. [CrossRef]
    33. Devereaux P, Bhandari M, Clarke M, Montori V, Cook D, Yusuf S, et al. Need for expertise based randomised controlled trials. BMJ 2005 Jan 8;330(7482):88 [FREE Full text] [CrossRef] [Medline]
    34. Besser H, Bonn M. Impact of distance independent education. J Am Soc Inf Sci 1996 Nov;47(11):880-883. [CrossRef]
    35. Wells R. Computer-mediated Communication for Distance Education: An International Review of Design, Teaching, and Institutional Issues. Pennsylvania: American Center for the Study of Distance Education; 1992.
    36. Phelps RH, Wells RA, Ashworth RL, Hahn HA. Effectiveness and costs of distance education using computer‐mediated communication. American Journal of Distance Education 1991 Jan;5(3):7-19. [CrossRef]
    37. Rumble B. Online costs: interactivity at a price. In: Mayson R, Kaye A, editors. Communication, Computers and Distance Education. Oxford: Pergamon Press; 1989.
    38. Feuer MJ, Towne L, Shavelson RJ. Scientific Culture and Educational Research. Educational Researcher 2002 Nov 01;31(8):4-14. [CrossRef]
    39. Levin J. What to do about educational research's credibility gaps? Issues in Education 1999;5(2):177-229. [CrossRef]


    Abbreviations

    BL: blended learning
    EMB: evidence-based medicine
    F2F: face-to-face
    ICER: incremental cost-effectiveness ratio
    MD: mean difference
    PV: present value
    QASE: quality-adjusted students educated
    RCT: randomized controlled trial


    Edited by G Eysenbach; submitted 12.02.15; peer-reviewed by S Sadasivan, D Cook; comments to author 23.03.15; accepted 08.07.15; published 21.07.15

    ©Stephen Maloney, Peter Nicklen, George Rivers, Jonathan Foo, Ying Ying Ooi, Scott Reeves, Kieran Walsh, Dragan Ilic. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 21.07.2015.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.