Reliability of the Grading of High-School Work in English

Abstract

I. Problem and method of investigation.-The reliability of the school's estimate of the accomplishment and progress of pupils is of large practical importance. For, after all, the marks or grades attached to a pupil's work are the tangible measure of the result of his attainments, and constitute the chief basis for the determination of essential administrative problems of the school, such as transfer, promotion, retardation, elimination, and admission to higher institutions; to say nothing of the problem of the influence of these marks or grades upon the moral attitude of the pupil toward the school, education, and even life. The recent studies of grades have emphatically directed our attention to the wide variation and the utter absence of standards in the assignment of values. Dearborn pointed out in his investigation' the large inequalities in the standards of grading employed by different teachers. Of two instructors in the same department one gave 43 per cent of his students the grade of "excellent" and to none the grade of "failure," whereas the other gave to none of his students the grade of "excellent" and to 14 per cent the grade of "failure." The wide difference in this instance is no doubt due in part to a difference in the students and in the nature of the work, but largely to a difference in the standards of marking.In order to determine precisely the personal equation in evaluating the work of pupils, it is necessary to eliminate all other causes of variation. The mere comparison of marks assigned by different teachers to their classes will not reveal this personal factor stripped of the other variable elements, such as difference in amount and kind of work covered by the class, emphasis upon different topics, 'W. F. Dearborn, School and University Grades (Bulletin of the University of Wisconsin, No. 368), p. 57. 442This content downloaded from 054.245

Pass/Fail and Discretionary Grading: A Snapshot of Their Influences on Learning

“…In 1912, Starch and Elliot's classic study examining how 147 high school English teachers assigned grades to two identical student papers revealed marks ranging from 50% to 90% for the same paper [11]. Later, in 2011, Brimi replicated the study and obtained strikingly similar results in that 73 high school English teachers assigned marks from 50% to 96% for the same paper [12].…”

The impact on stakeholder confidence of increased transparency in the examination assessment process

“…This is surprising given that moderationthe principal component of most institutional assessment processes designed around a system of task repetition -has consistently been shown to produce significant disparities between assessors (Hartog and Rhodes 1935;Meadows and Billington 2005;Starch and Elliot 1912). The purpose of moderation is to ensure that marks awarded are reliable and appropriate regardless of who is doing the marking (Sadler 2013) but it often fails to meet this core objective.…”

When precedence sets a bad example for reform: conceptions and reliability of a questionable high stakes assessment practice in Norwegian universities

“…Therefore, the findings reported in Raaheim's study may have potentially inflated the impression of agreement among raters by representing linear variations in grading trends rather than variations in actual scores. Given the nearly century-old findings that already demonstrated low inter-grader reliability on essay exams (Starch & Elliott, 1912, 1913a, and the later findings of scepticism among faculty about the reliability of subjective grading processes in general (e.g., Ashburn, 1938;Goulden & Griffin, 1997), one would expect considerable variance in Raaheim's (2000) data. Therefore, this re-analysis of the original data was done in order to capture the degree of variance among raters around critical cut points for students and to consider the human effect of that variance.…”

“…The research has taught us that there are numerous factors to control for, such as discipline (Starch & Elliott, 1912, 1913aAshburn, 1938), the nature of the questions asked (Starch & Elliott, 1912;Ashburn, 1938;Michael et al, 1980;Ebel & Frisbie, 1986), the restricted range of the content examined (Ebel & Frisbie, 1986), the length of the answers provided (e.g., Ackerman & Smith, 1988), the overall form and appearance of the answer (Starch & Elliott, 1913a), the background of the student (e.g., Entwistle & Entwistle, 1992), the background of the exam reader (e.g., Raaheim, 2000), the criteria for grading (e.g., Mehrens & Lehmann, 1991) and the subjectivity of the scoring (Ebel & Frisbie, 1986). Since graders have little control over the questions students are asked, the lengths and appearances of the answers written and the background of the students being examined, this research focuses on the exam readers themselves and the implicit standards they use when grading exams.…”