Leveraging Cognitive Diagnosis to Improve Peer Assessment in MOOCs

Abstract

A major challenge faced by popular massive open online courses (MOOCs) is the assessment of large-scale open-ended assignments submitted by students. Recently, peer assessment has become a mainstream paradigm that helps grade open-ended assignments on a large scale. In peer assessment, students also become graders in grading a small number of their peers' assignments, and the peer grades are then aggregated to predict a true score for each assignment. The collected peer grades are usually inaccurate because graders have different reliabilities and biases. To improve accuracy, several probabilistic graph models have been proposed to model the reliability and bias of each grader. However, none of these models consider graders' competency information in the assignments to be graded, which has been found to be very effective. We propose two new probabilistic graph models to improve the accuracy of cardinal peer assessments based on the well-accepted cognitive diagnosis technique. Specifically, the cognitive diagnosis model DINA is applied to determine grader competency based on historical tests or assignments. Then, this information is used to optimize the modeling of grader reliability in each of the proposed models. Moreover, an effective model inference algorithm is proposed to infer true scores of assignments. Experimental results based on real world datasets show that the two proposed models outperform state-of-the-art models and that consideration of grader competency contributes to improved score estimation.

Tracing Knowledge States through Student Assessment in a Blended Learning Environment

“…Additionally, equations ( 4) and ( 5) are used in the evaluation stage to calculate RMSE and MAE. According to the equation, 𝑐 𝑖 represents the ground truth score and 𝑝 𝑖 represents the predicted score [25]. The RMSE provides a summary of the model's prediction error, with a lower value indicating better performance [20].…”

Towards a UTAUT model for acceptance of massive open online courses (MOOCs)

“…The significant increase in the application of IT across educational establishments is causing a pivotal shift in the way learners learn and teachers teach, with this process a continuous one that is designed in mind of collaboration and ensuring high-quality courses. Recent announcements made by top universities in their changes of direction to the provision of education in new and innovative ways, namely through MOOCs not only has been successful in garnering attention from students and academics in higher education, but also from learners and teachers in the K-12 environment, in home schooling, or otherwise in the conventional classroom setting (Xu et al, 2021). Through the provision of MOOCs, ways of teaching are becoming more diverse and are shifting away from conventional methods, with the teacher not necessarily serving as the holder and communicator of knowledge, but instead with learners interacting with peers and students, and being able to access information and resources prior to entering a classroom environment.…”

On the Supervision of Peer Assessment Tasks: An Efficient Instructor Guidance Technique

“…Shalem et al [43], considering interaction with multiple instructors, modeled the student's ability, the instructor's competence, and the question difficulty. Others extended the Piech et al [5] key models to use assessor's deviation grades [44], ordinal grades [45] or understanding ordinal grades as censored data [46], as well as to account for other factors such as social interactions between students [47], cognitive diagnosis-derived student competency [48], or student effort expended [46]. We study the first and third models of [5] in the context of our methodology (explained in Sect.…”