Rubric Development And Inter Rater Reliability Issues In Assessing Learning Outcomes

Abstract

This paper describes the development of rubrics that help evaluate student performance and relate that performance directly to the educational objectives of the program. Issues in accounting for different constituencies, selecting items for evaluation, and minimizing time required for data analysis are discussed. Aspects of testing the rubrics for consistency between different faculty raters are presented, as well as a specific example of how inconsistencies were addressed. Finally, a considerat ion of the difference between course and programmatic assessment and the applicability of rubric development to each type is discussed.

An Assessment of Reliability and Validity of a Rubric for Grading APA-Style Introductions

“…Chance agreement given four scoring levels and three graders would be .34 and .06 by the liberal and conservative definitions, respectively. Newell et al (2002) found comparable levels of agreement for three graders using a rubric for grading students' solutions of chemical engineering problems, a task that was not writing based. The rubric developed by Newell et al also had four scoring levels within each dimension.…”

“…Agreement was defined liberally as all scores assigned for a dimension by the three graders being within 1 point of one another. These criteria are accepted in the measurement literature (Tinsley & Weiss, 2000) and have been applied in past studies of interrater agreement for grading rubrics (Newell et al, 2002). Agreement on total overall score out of 24 possible points (8 dimensions × 3 points maximum for each) for the 40 papers was also calculated and is described in the results.…”

“…Without specific grading instructions, one grader might emphasize grammar, for example, whereas another might emphasize content. A well-developed rubric guides graders in placing the desired emphasis on specific, uniform criteria, so that the role of subjective opinions is minimized (Newell, Dahm, & Newell, 2002). Training of graders usually is necessary to maximize interrater agreement (Zimmaro, 2004).…”

Incentivizing Multiple Revisions Improves Student Writing Without Increasing Instructor Workload

“…For example, Stellmack, Konheim-Kalkstein, Manor, Massey, and Schmitz (2009) found low interrater agreement (agreement between reviewers in .37 of the scores they assigned) for graders who developed and refined a grading rubric over several months. Newell, Dahm, and Newell (2002) also reported comparably low interrater agreement (.47 proportion of agreement measured in the same way as Stellmack, Konheim-Kalkstein, Manor, Massey, & Schmitz, 2009) in the grading of student writing with a rubric. 1 Indeed, subjectivity and low interrater agreement in evaluating scientific writing are implicitly acknowledged in the peer-review process when an editor seeks reviews from multiple reviewers.…”

Refining A Critical Thinking Rubric For Engineering

“…A decision was made to have a four-level scale for the rubric, which is consistent with other university-wide holistic rubrics & minimizes the tendency to rate in the middle of odd number level scales. 33,38 Listing includes a description of the various performance levels that are used to write the dimension descriptions. 31 Engineering faculty reviewed the Paul-Elder critical thinking framework to identify key Elements of Thought and Universal Intellectual Standards that would be applicable across engineering courses.…”