This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Massive open online courses (MOOCs), as originally conceived, promised to provide educational access to anyone with an internet connection. However, the expansiveness of MOOC education has been found to be somewhat limited. Nonetheless, leading universities continue to offer MOOCs, including many in the health sciences, on a number of private platforms. Therefore, research on online education must include thorough understanding of the role of MOOCs. To date, studies on MOOC participants have focused mainly on learners’ assessment of the course. It is known that MOOCs are not reaching the universal audiences that were predicted, and much knowledge has been gained about learners’ perceptions of MOOCs. However, there is little scholarship on what learners themselves gain from participating in MOOCs.
As MOOC development persists and expands, scholars and developers should be made aware of the role of MOOCs in education by examining what these courses do offer their participants. The objective of this qualitative synthesis of a set of MOOC evaluation studies was to explore outcomes for MOOC learners, that is, how the learners themselves benefit from participating in MOOCs.
To explore MOOC learners’ outcomes, we conducted a qualitative synthesis in the form of a deductive thematic analysis, aggregating findings from 17 individual studies selected from an existing systematic review of MOOC evaluation methods. We structured our inquiry using the Kirkpatrick model, considering Kirkpatrick levels 2, 3, and 4 as potential themes in our analysis.
Our analysis identified six types of Kirkpatrick outcomes in 17 studies. Five of these outcomes (learning/general knowledge, skills, attitudes, confidence, and commitment) fit into Kirkpatrick Level 2, while Kirkpatrick Level 3 outcomes concerning behavior/application were seen in four studies. Two additional themes were identified outside of the Kirkpatrick framework: culture and identity outcomes and affective/emotional outcomes. Kirkpatrick Level 4 was not represented among the outcomes we examined.
Our findings point to some gains from MOOCs. While we can expect MOOCs to persist, how learners benefit from the experience of participating in MOOCs remains unclear.
When the first massive open online course (MOOC) was offered in 2008, the MOOC format—free, online, and open to anyone with an internet connection—was touted as revolutionary for its potential to democratize access to educational opportunities due to its theoretically universal availability [
Learning is a complex phenomenon that can be described from different perspectives. Understanding learning is about understanding not only learning processes but also the conditions that influence—and are influenced by—the learning process [
A number of systematic reviews have examined MOOCs [
Another systematic review by Rowe et al [
In their recent systematic review, Alturkistani et al [
Thus, despite their persistence, MOOCs have not lived up to the early expectation that they would allow widespread, nearly universal access to education. For example, there is consistent evidence that learners who use MOOCs, and indeed those who are more likely to complete them, are generally more educated and affluent [
We conducted a qualitative synthesis in the form of a deductive thematic analysis, aggregating findings from individual studies, to explore MOOC learners’ outcomes. The datasets used and analyzed during the current study are available from the first author on reasonable request. To structure our inquiry, we relied on a commonly used framework for evaluating learning with applications in multiple learning and training settings: the Kirkpatrick model [
The studies in the current synthesis derive from Alturkistani et al’s systematic review of MOOC evaluation methods [
Specifically, Alturkistani et al’s “learning outcomes and experiences” subcategory was the basis for the current synthesis, as we looked at what learners gain from the experience of participating in MOOCs. This subcategory included 21 studies. We reviewed each paper in this category for findings that included learners’ outcomes
More specifically, in this qualitative synthesis, we performed a deductive thematic analysis [
Outcomes that could not be matched with the Kirkpatrick levels were set aside for a separate inductive thematic analysis, which is presented as “Outcomes beyond Kirkpatrick.”
Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram: systematic review (A) for a synthesis paper on MOOC learning outcomes (B). Modified from Alturkistani et al [
Our analysis resulted in six types of outcomes. These are summarized in
Our deductive analysis showed that 15/16 (94%) of the examined studies included one or more outcomes corresponding to Kirkpatrick Level 2. Thus, the Learning theme here incorporates concepts such as knowledge, skill, attitude, confidence, and commitment. Each subtheme is presented using the identified data and illustrated with supporting quotations.
Most of the Level 2 outcomes we identified were scores or survey items that assessed knowledge in some form. For example, in their MOOC on new media in teaching and learning, Chen et al [
Outcomes of MOOC studies framed by Kirkpatrick Level 2 or Level 3.
Kirkpatrick level, subtheme, and study | Data collection | Data analysis | Outcome variables | Outcome findings | ||
|
||||||
|
|
|||||
|
|
Chen et al (2015) [ |
Scores on quizzes and final paper | Inferential statistics | Possible “Excellent Paper,” “Excellent Participation,” and “Excellent Group Member” awards | Learners received these awards if they fulfilled the criteria |
|
|
Konstan et al (2015) [ |
Three-part longitudinal design: precourse, postcourse, and 5-month follow-up “knowledge tests” and surveys | Inferential statistics; qualitative analysis | Assessed knowledge of recommender systemsa | Gains in knowledge and 5-month retention of acquired knowledge |
|
|
Jacquet et al (2018) [ |
LMSb data; pre-MOOC and post-MOOC knowledge tests | Inferential statistics | Score on knowledge test | Increased knowledge score from pretest to posttest |
|
|
Liang et al (2014) [ |
Assessments: quizzes and homework | Inferential statistics | Average assessment score | Increase in assessment score related to degree of participation |
|
|
Cross (2013) [ |
Precourse and postcourse surveys; LMS | Descriptive statistics | Knowledge: “novice” to “expert”a | Increase in knowledge |
|
|
Colvin et al (2014) [ |
Normalized gain between pretests and posttests in introductory physics; “ability” based on test items attempted, analyzed with Item Response Theory (IRT) | Inferential statistics | Comparison of pre-MOOC and post-MOOC physics knowledge and “ability” | Learning (measured via posttest score) across several cohorts identified using IRT |
|
|
MacKay et al (2016)[ |
Precourse and postcourse assessments of animal welfare knowledge | Inferential statistics | Scores on animal welfare knowledge assessment |
Increased scores |
|
|
|||||
|
|
Brunton et al (2017)[ |
Weekly Likert scale quizzes during the MOOC: “individual digital readiness tools” and postcourse quiz | Descriptive statistics | Preparedness for online learninga | Self-assessed changes in preparedness for online learning |
|
|
Rubio (2015)[ |
Precourse and postcourse comprehensibility ratings | Inferential statistics | Spanish comprehensibility (language pronunciation) | Increased comprehensibility in postcourse ratings |
|
|
Stephens and Jones (2014) [ |
Precourse and postcourse surveys with mostly open-ended items | Content analysis | Skills discoverya | Technological skills |
|
|
Liu et al (2014) [ |
End-of-course surveys (Likert scale and open-ended); email interviews | Descriptive and thematic analysis (focused coding) | Three things students learneda | Skills in data visualization, critiquing, and creating infographics |
|
|
|||||
|
|
Alturkistani et al (2018) [ |
Case studies; interviews | Thematic analysis | Learning achievement; use of information in the workplacea | Intention to apply knowledge |
|
|
|||||
|
|
MacKay et al (2016) [ |
Multiple-choice quizzes; confidence and attitude surveys (mostly Likert scale) | Inferential statistics | Change in attitudes; certificate of achievement for completiona | Change in attitude |
|
|
|||||
|
|
Hossain et al (2015) [ |
Ten-point scale; confidence-to-treat | Inferential statistics | Confidence to treat spinal cord injurya | Gains in confidence |
|
|
Cross (2013) [ |
Precourse/postcourse survey; LMS | Descriptive statistics | Confidence to apply learninga | Gains in confidence |
|
|
Mackness et al (2013) [ |
Interviews (face-to-face and email) and focus groups; assessment of microteaching | Qualitative case study approach | Confidence to participate in social learning environmentsa | Gains in confidence |
|
|
Lei et al (2015) [ |
Pre-MOOC and post-MOOC surveys; forum threads | Sentiment analysis | Identity and confidencea | Confidence in work; confidence to inspire |
|
|
Milligan and Littlejohn (2014) [ |
Interviews mid-MOOC | Qualitative analysis | Changes in practicea | Confidence about practices on the job |
|
||||||
|
|
|||||
|
|
Milligan and Littlejohn (2014) [ |
Survey and interview | Qualitative analysis | Application of learning in professional practicea | Integrating new understanding in practice |
|
|
Lei et al (2015) [ |
pre-MOOC and post-MOOC surveys; forum threads | Sentiment analysis | Effects on learners and communitya | Bringing knowledge back to community |
|
|
Cross (2013) [ |
Precourse/postcourse survey; LMS | Descriptive statistics | Changes in practicea | Implementation of tools in course design |
|
|
Konstan et al (2015) [ |
Follow-up interview and survey | Inferential statistics | Application of new recommender system skillsa | Application of systems at work, school, business |
aIncludes a self-report.
bLMS: learning management system.
We found several examples of skill outcomes, including self-assessed preparedness (readiness for online education) [
Other Level 2 outcomes were commitment, as shown through intention to apply knowledge [
They also gain the confidence to attend and contribute to live synchronous sessions, to openly share their work and ideas, and to cooperate and/or collaborate in social networking environments. “They shift from being consumers to producers.”
In their MOOC on Asian vernacular architecture, Lei et al [
It is through learning that I have gained the most confidence, in my identity and in my work. And I hope that this course would be the one of many stepping stones towards me being able to help inspire and nurture future generations….
Using a clinical trials MOOC, Milligan and Littlejohn [
Our analysis found 4/21 studies (19%) with evidence of Level 3 outcomes. Level 3 includes application via critical behaviors plus the presence of outcomes that make it more likely that people will retain and apply what they have learned in a given setting (the abovementioned catalysts for application or “required drivers”).
In addition to effects on confidence (Level 2), Milligan and Littlejohn [
Lei and colleagues [
This course helped me to see the significance of the collapsed houses, temples, shrines, monuments and courtyards in a different angle which otherwise I would not have been able to see…I have already started contributing my knowledge with the local community as we come together to rebuild what has been destroyed.
Cross [
In this qualitative synthesis, we did not find any data congruent with Kirkpatrick Level 4, which includes outcomes and “leading indicators.”
Not all of the outcomes described in the studies are congruent or align well with the Kirkpatrick framework; hence, we present these outcomes separately here. After our inductive thematic analysis, we identified two themes among these outcomes: “culture and identity” outcomes and “affective/emotional” outcomes. Culture and identity outcomes included “insights about themselves through personal reflection about their learning styles, professional practices, and the ways they view the world” [
In this qualitative analysis, we explored the benefits that MOOCs in a broad range of subjects offer their participants. We synthesized the types of outcomes reported in a set of MOOC studies, including but not limited to outcomes that assess learning in some way. Using the Kirkpatrick model as a framework, the most prominent findings were that most of the MOOCs described in the included studies only had outcomes that could be categorized as Kirkpatrick Level 2. Kirkpatrick Level 3 outcomes were also represented, although these were not as common as Level 2 outcomes. We did not observe any Kirkpatrick Level 4 outcomes in the data we analyzed. If a MOOC were to aim for or result in Level 4 outcomes, we would expect to see changes at the organizational level. This might reveal itself in the form of implemented changes in policy in a health care setting after a group of managers participated in a policy MOOC, or in the case of higher education, a change in pedagogical training for educators after a MOOC was attended by several faculty members. Our complementary analysis of outcomes that did not align with Kirkpatrick yielded two additional themes.
Previous research has shown that students generally perform better in face-to-face courses than in online courses [
MOOCs were also expected to foster and build social networks. However, in reality, the amount of interaction among MOOC participants is often limited, and a small proportion of learners are usually responsible for most of this interaction. This finding was reinforced by the studies we examined [
Using a well-known model to frame and lens our findings, in this study, we explored one understudied aspect of MOOCs that provides a view of what learners can gain from MOOCs. The richness of data using an in-depth secondary analysis of a small number of studies from a systematic review with broad subject matter, combined with frequent debriefing sessions and investigator triangulation, enhanced the credibility of the findings. We argue that qualitatively synthesizing existing data in an attempt to make sense of contextually and methodologically diverse findings is an important contribution to the scholarly literature. There are also some limitations to this study. Synthesizing both quantitative and qualitative data is a daunting task, as these data derive from very different paradigms. Thus, an important factor limiting the applicability of our findings is the problem with extracting results from eclectic and dissimilar studies, including qualitative and quantitative methods and grey literature, and attempting to contrast and compare them. The findings should thus be interpreted with due caution in light of this fact. Further, as our work builds on a previous review, we included only studies that were included therein. This may leave out some relevant studies, despite the rigorous inclusion criteria of the previous review. Finally, despite the frequent scholarly use of the Kirkpatrick framework, there are some inherent limitations to the model that also have implications for this work. It has been argued that the four-level model depicts an oversimplified view of learning and training effectiveness that does not take individual or contextual influences into account in the evaluation of the learning that occurs [
Our findings point to some gains from MOOCs, and while we can expect MOOCs to persist, how learners benefit from the experience of participating in these courses remains unclear. This is especially true when comparing MOOCs to other learning modes, as evidenced by the comparative studies included in our sample. In our study, we looked for gains or benefits to MOOC learners in all subject areas, and we used the Kirkpatrick framework to explore what learners might gain. From a diverse set of studies, we found outcomes that included changes in knowledge, skills, attitude, and confidence as well as changes in behavior, increased excitement about a subject, and effects on cultural identity as a result of MOOC participation. Thus, beyond outcomes that can be classified as “learning,” such as increased knowledge or skill, it does appear that MOOCs provide some value for participants via the gains described above.
In contrast to systematic reviews of MOOC research, we carried out a deeper qualitative analysis of a set of studies from one systematic review that looked only at MOOC evaluation methods. Thus, as an extension of Alturkistani et al [
Description of the analytic procedure for the qualitative synthesis.
connectivist massive open online course
learning management system
massive open online course
open online course
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
extended massive open online course
We would like to thank Abrar Alturkistani, MPH and Edward Meinart, PhD for collaboration on the EIT Health project that inspired this paper. We also thank Hanna Augustsson, PhD for her expert review of our application of the Kirkpatrick model. All conclusions are the authors’ own. This work was partially funded by EIT Health (Grant 18654).
ERB, TS, and PJP contributed to the conception, study design, data collection, analysis and interpretation, and drafting and critical revision of the manuscript. All authors approved the final version of the manuscript.
None declared.