Assessment that matters

Abstract

Learner-centered pedagogy highlights active learning and formative feedback. Instructors often incentivize learners to engage in such formative assessment activities by crediting their completion and score in the final grade, a pedagogical practice that is very relevant to MOOCs as well. However, previous studies have shown that too many MOOC learners exploit the anonymity to abuse the formative feedback, which is critical in the learning process, to earn points without effort. Unfortunately, limiting feedback and access to decrease cheating is counter-pedagogic and reduces the openness of MOOCs. We aimed to identify and analyze a MOOC assessment strategy that balances this tension between learner-centered pedagogy, incentive design, and reliability of the assessment. In this study, we evaluated an assessment model that MITx Biology introduced in a MOOC to reduce cheating with respect to its effect on two aspects of learner behavior -the amount of cheating and learners' engagement in formative course activities. The contribution of the paper is twofold. First, this work provides MOOC designers with an 'analytically-verified' MOOC assessment model to reduce cheating without compromising learner engagement in formative assessments. Second, this study provides a learning analytics methodology to approximate the effect of such an intervention. CCS CONCEPTS• Applied computing → E-learning; • Unsupervised learning → Anomaly detection.

Defining and Measuring Completion and Assessment Biases with Respect to English Language and Development Status: Not All MOOCs are Equal

“…Our IRT methodology resulted in a unique group of learners that represent the core learning group in the course-run hence allowing to shed some light on the relation between ability estimate and completion rate. Assessment serves both learning and measurement purposes, and may be the single most important factor affecting learning in MOOCs [17,18,16]. If learners from developing countries face increased difficulty during learning, they will be more likely to drop-out, although the ones that "survive", do not "pay" in a decreased estimate of their ability, usually, and sometimes excel over the learners from developed countries.…”

“…Resource-oriented approaches that focus on making videos -the primary learning resource in MOOCs [4] -more adapted to ELLs, were presented in [28,29]. However, this issue was not explored with relation to assessment, which plays a key role in the learning process of MOOC students [17,16].…”

“…The relation between course design and learning in MOOCs has been mainly studied through between-subjects, within-course analysis, typically in order to optimize learning by identifying causal links between behavior and learning using data mining methods (e.g., [34,17,6,35]). Within these, several studies highlighted the role of on-going assessment for both measuring (summative) and promoting (formative) learning in MOOCs, as well as for supporting analytics-based design [16,17,36]. However, there seems to be scarcity of cross-course studies that relate course factors and learning achievements.…”

“…There are various approaches for measuring learner performance in MOOCs, each raising different pedagogic and methodological issues [24,44,16]. We use two measures: course completion, and ability estimate that is based on learner responses to the course assessment.…”

“…In many successful MOOCs (and specifically, the MITx MOOCs that we study), formative assessment also serves for summative purposes. This pedagogy provides learners with resources and incentives for active learning [16], which is found to be highly effective in MOOCs [17,18]. So, while bias in summative assessment might indicate that students learned less, bias in formative assessment may indicate that the opportunities for learning that these learners received were less adequate to their needs.…”