Reliability and Validity of an Introductory Physics Problem-Solving Grading Rubric

Abstract

We have developed and validated a rubric for the assessment and scaffolding of problem-solving process in introductory physics courses via an iterative approach. The current version of the rubric consists of eight criteria based on research in expert-like problem solving practice and aspects of Cooperative Group Problem Solving (CGPS) pedagogy. In contrast to recent work on problem-solving assessment for use in research and curriculum development, this rubric was specifically designed for instructor use in the assignment of grades and for student use as a scaffold. This means that the rubric can be used within group problem-solving activities as a student support, formative assessment of individual work, and summative assessment, such as exams. For this study, the rubric was used to score N = 166 student solutions to 6 individually-assigned homework problems covering content in introductory mechanics in a course enrolling 32 students. Inter-rater and re-rater reliability was high for undergraduate Learning Assistant raters receiving only moderate training (approximately 4 hours). Factor analysis identified two factors that have been categorized as: (1) framing & defining, and (2) planning & execution. These factors align with our initial theory of the construct, suggesting evidence for criterion-related validity. Tau-equivalent reliability was found to be 0.76, and an item-total correlations test demonstrated all criteria correlations consistent with averaged behavior.