Bias in Student Ratings of Instruction: A Systematic Review of Research from 2012 to 2021

Abstract

Student ratings of instruction (SRI) are commonly used to evaluate courses and teaching in higher education. Much debate about their validity in evaluating teaching exists, which is due to concerns of bias by factors unrelated to teaching quality (Spooren et al., 2013). Our objective was to identify peer-reviewed original research published in English from January 1, 2012, to March 10, 2021, on potential sources of bias in SRIs. Our systematic review of 63 articles demonstrated strong support for the continued existence of gender bias, favoring male instructors and bias against faculty with minority ethnic and cultural backgrounds. These and other biases must be considered when implementing SRIs and reviewing results. Critical practices for reducing bias when using SRIs include implementing bias awareness training and avoiding use of SRIs as a singular measure of teaching quality when making decisions for teaching development or hiring and promotion.

Quantifying Careless Responses in Student Evaluation of Teaching and Justifying Removal for Data Validity

“…Research has consistently detected biases in SET. For instance, Stoesz et al (2022) systematically reviewed the articles published between 2012 and 2021 and concluded that biases existed against female and minority faculty. Feistauer and Richter (2018) reported that the effect of likability and prior subject interest exerted significant bias on student evaluation.…”

“…For example, some research has confirmed the validity of SET (Marsh, 1987). Other research has also demonstrated various sources of invalidity of SET survey data, including bias (i.e., the deviation from the truth), the halo effect (i.e., the tendency to assign similar ratings), and other factors unrelated to teaching (e.g., Stoesz et al, 2022). Evaluation survey data validity impacts evaluation feedback's utility (i.e., usefulness).…”

Modeling unit non-response and validity of online teaching evaluation in higher education using generalizability theory approach

“…Due to the high-stakes nature of appraisal data, biases that threaten the validity of the information provided have been studied extensively across geographical boundaries ( Spooren et al, 2013 ; MacNell et al, 2015 ; Hornstein, 2017 ; Quansah, 2020 ; Kreitzer and Sweet-Cushman, 2021 ; Quansah, 2022 ; Stoesz et al, 2022 ). These previous studies have stressed that the sources of variation in students’ appraisal of teaching are attributed to raters (i.e., students), items, occasions, course types, and teacher characteristics unrelated to teaching.…”