Determining the Consistency of Student Grading in a Hybrid Business Course using a LMS and Statistical Software

Abstract

Extant literature asserted that peer assessments improved learning but only a few studies had addressed the student rating consistency issue and not had evaluated this within a Learning Management System (LMS). The researcher explored how to conduct peer assessments in Moodle LMS for a hybrid-mode business course (N=90 students) which required 270 (25-page) reports. Rater agreement statistical theory was applied to test the consistency of student peer assessments. The resulting coefficient was 0.79 and statistically significant. A pair-wise t-test and a nonparametric Mann Whitley u-test confirmed there was no significant difference between the student ratings and the scores given by the professor on the same assignments. Nonparametric Spearman correlation was very high between each student rating coefficient as compared to the score given in Moodle, which indicated the LMS was reliable in providing a grade for the quality of student peer assessment. A protocol was developed using SPSS to calculate a peer assessment inter-rater agreement consistency indicator since this was not available in Moodle.

Hybrid and Online Peer Group Grading: Adding Assessment Efficiency While Maintaining Perceived Fairness

“…The results of this study show no significant difference in group grades based on peer group grading or instructor grading. These results for peer group grading extend the research by Strang (2013), where peer grades and instructor grades in a hybrid course were not significantly different for individual student submissions. These results also build on a study by Avery (2014), that found highly correlated instructor and peer-to-peer assessment of class participation.…”

“…Kulkarni et al (2013) found that providing students with well-specified dimensions in rubrics for peer assessment enhances agreement between student generated scores and instructor feedback. This approach was shown to be effective in providing consistency in student peer and instructor grading at the individual level (Strang, 2013). Applying the structured approach to peer group grading overcomes that variability in academic preparedness and also reduces the instructor necessity to average, or in some way tabulate individual scores for each peer group.…”

Technology-enhanced pre-instructional peer assessment: Exploring students’ perceptions in a Statistical Methods course

“…The major concern in peer assessment is its validity as well as reliability (Cho, Schunn, & Wilson, 2006). Topping (1998) found disagreement on the degree of validity and reliability of peer assessment on his review, some studies report high validity and reliability (Haaga, 1993;Stefani, 1994;Strang, 2013), and the others report otherwise (Cheng & Warren, 1999;Mowl & Pain, 1995). However, the issues regarding validity and reliability can be reduced by providing the students with assessment rubrics (Hafner & Hafner, 2003;Jonsson & Svingby, 2007) since it makes expectations and criteria explicit.…”