Massive Open Online Course Evaluation Methods: Systematic Review

Abstract

Background Massive open online courses (MOOCs) have the potential to make a broader educational impact because many learners undertake these courses. Despite their reach, there is a lack of knowledge about which methods are used for evaluating these courses. Objective The aim of this review was to identify current MOOC evaluation methods to inform future study designs. Methods We systematically searched the following databases for studies published from January 2008 to October 2018: (1) Scopus, (2) Education Resources Information Center, (3) IEEE (Institute of Electrical and Electronic Engineers) Xplore, (4) PubMed, (5) Web of Science, (6) British Education Index, and (7) Google Scholar search engine. Two reviewers independently screened the abstracts and titles of the studies. Published studies in the English language that evaluated MOOCs were included. The study design of the evaluations, the underlying motivation for the evaluation studies, data collection, and data analysis methods were quantitatively and qualitatively analyzed. The quality of the included studies was appraised using the Cochrane Collaboration Risk of Bias Tool for randomized controlled trials (RCTs) and the National Institutes of Health—National Heart, Lung, and Blood Institute quality assessment tool for cohort observational studies and for before-after (pre-post) studies with no control group. Results The initial search resulted in 3275 studies, and 33 eligible studies were included in this review. In total, 16 studies used a quantitative study design, 11 used a qualitative design, and 6 used a mixed methods study design. In all, 16 studies evaluated learner characteristics and behavior, and 20 studies evaluated learning outcomes and experiences. A total of 12 studies used 1 data source, 11 used 2 data sources, 7 used 3 data sources, 4 used 2 data sources, and 1 used 5 data sources. Overall, 3 studies used more than 3 data sources in their evaluation. In terms of the data analysis methods, quantitative methods were most prominent with descriptive and inferential statistics, which were the top 2 preferred methods. In all, 26 studies with a cross-sectional design had a low-quality assessment, whereas RCTs and quasi-experimental studies received a high-quality assessment. Conclusions The MOOC evaluation data collection and data analysis methods should be determined carefully on the basis of the aim of the evaluation. The MOOC evaluations are subject to bias, which could be reduced using pre-MOOC measures for comparison or by controlling for confounding variables. Future MOOC evaluations should consider using more diverse data sources and data analysis methods. International Registered Report Identifier (IRRID) RR2-10.2196/12087

Benefits of Massive Open Online Course Participation: Deductive Thematic Analysis

“…The studies in the current synthesis derive from Alturkistani et al’s systematic review of MOOC evaluation methods [ 21 ]. Their review included studies from 2008 to 2018 that focused primarily on MOOC evaluation and studies that reviewed or applied MOOC evaluation methods.…”

“…This is in line with the epistemological stance of this review. The complete search strategy and further details of the source review [ 21 ] can be found in [ 18 ]. Alturkistani et al [ 21 ] identified 3275 records; after a review procedure, the final review included 33 studies.…”

“…We did not include measures of engagement, motivation, completion, or attrition in our analysis unless they were clearly tied to the outcomes for learners. In an additional step intended to capture all learner outcomes, we examined Multimedia Appendix 3 in Alturkistani et al’s review [ 21 ], which included all 33 studies. As a result of this review, we excluded 12 studies that did not include clear outcomes for learners ( Figure 1 ), which left 21 studies for our analysis.…”

“…In their recent systematic review, Alturkistani et al [ 21 ] also added to the discourse on MOOC evaluation methods. Alturkistani et al identified three “evaluation-focused categories” among the studies they reviewed: learner-focused, teaching-focused, and MOOC-focused [ 21 ]. We approached this review as a jumping-off point to further synthesize understanding of MOOC learner outcomes.…”

“…We approached this review as a jumping-off point to further synthesize understanding of MOOC learner outcomes. Here, we unpack the learner-focused category in [ 21 ] and, more specifically, the “learning outcomes and experience” subcategory to investigate the learner outcomes for the included MOOCs. In our study, “learner outcomes” are direct statements that describe the knowledge, skills, and attitudes that learners have demonstrated or are expected to reliably demonstrate when successfully completing a course.…”