Group Optimization to Maximize Peer Assessment Accuracy Using Item Response Theory and Integer Programming

Abstract

With the wide spread large-scale e-learning environments such as MOOCs, peer assessment has been popularly used to measure the learner ability. When the number of learners increases, peer assessment is often conducted by dividing learners into multiple groups to reduce the learner's assessment workload. However, in such cases, the peer assessment accuracy depends on the method of forming groups. To resolve that difficulty, this study proposes a group formation method to maximize peer assessment accuracy using item response theory and integer programming. Experimental results, however, have demonstrated that the method does not present sufficiently higher accuracy than a random group formation method does. Therefore, this study further proposes an external rater assignment method that assigns a few outside-group raters to each learner after groups are formed using the proposed group formation method. Through results of simulation and actual data experiments, this study demonstrates that the proposed external rater assignment can substantially improve peer assessment accuracy.

Adaptive learning module for a conversational agent to support MOOC learners

“…IRT has already been used in educational contexts to adapt the questions asked to the learners, for example with parametric exercises (Muñoz-Merino, Novillo, Delgado Kloos, 2018). In the case of MOOCs, IRT has been used to avoid cheating on graded tests (Alexandron et al, 2016;Meyer & Zhu, 2013), or to improve peer assignment in peer assessment activities (Uto, Nguyen, & Ueno, 2019;Uto, & Ueno, 2018). However, and to the best of the authors' knowledge, no research work has been published regarding the adaptation of questions through IRT in conversational agents.…”

Accuracy of performance-test linking based on a many-facet Rasch model

“…Concretely, the data can be defined as where is the rating categories, and x i j r = − 1 indicates missing data. Missing data occur in actual performance assessments because few raters are generally assigned to individual evaluation targets to lessen the scoring burden (Engelhard, 1997 ; Eckes, 2015 ; Ilhan, 2016 ; Uto et al, 2020 ). A typical rater assignment strategy is the rater-pair design (Eckes, 2015 ), which assigns two raters to each evaluation target.…”

“…We assume the rate of missing data as a factor affecting linking accuracy because it affects parameter estimation accuracy (Uto, Duc Thien, & Ueno, 2020). Note that missing data occur in practice because few raters are generally assigned to individual evaluation targets to lessen raters' scoring burdens.…”

“…These findings suggest that the extent to which IRT-based performance-test linking requires common raters and tasks depends basically on the following factors: distributions of examinee ability and characteristics of raters and tasks, numbers of examinees, raters, and tasks, and rates of missing data. We assume the rate of missing data as a factor affecting linking accuracy because it affects parameter estimation accuracy (Uto, Duc Thien, & Ueno, 2020 ). Note that missing data occur in practice because few raters are generally assigned to individual evaluation targets to lessen raters’ scoring burdens.…”

التفاعل بین نمطی استراتیجیة تقییم الأقران (زوجی- مجموعات) ونمط الاستجابة (حرة- موجهة) فی بیئة تعلم قائمة على الویب، وأثره على خفض قلق الاختبارات الإلکترونیة وجودة التعلم لدى طلاب کلیة العلوم والآداب

“…. (Sukstrienwong, 2017;Haddadi et al, 2018;Bi, Mengyu et al, (Parsons, 2004) ‫رت‬ ‫دددددد‬ ‫وأشد‬ ، ‫ت‬ ‫الدراسددد‬ ‫عض‬ (Sridharan et al, 2019;Handayani & Genisa, 2019;Garcia-Souto, 2019;Babail & Adeh, 2019;Mohd et al, 2019) (Cheng & Hou,2015;Rosa et al, 2016;Sukstrienwong, 2017;Surahman et al, 2018;Mohd et al, 2019;Li & Grion, 2019;Wang et al, 2019;Cleves-Diaz, 2020;Wang et al, 2020) ‫ديمي‬ Chan, 2018;Haddadi et al, 2018;James et al, 2018;Bi et al, 2019;Babaii & Adeh, 2019;Fu, Lin & Hwang, 2019;Garcia-Souto, 2019;Handayani & Genisa, 2019;Sridharan et al, 2019;Uto et al, 2019) . (Sukstrienwong, 2017;Haddadi et al, 2018;Bi et al, Vanderhoven, E., Raes, A., Montrieux, H., Rotsaert, T., & Schellens, T. (2015).…”