Content bias in math testing: Items about money, food, and social interaction disadvantage students from low socioeconomic backgrounds

Abstract

In many countries, standardized math tests are important for achieving academic success. Here, we examine whether content of items, the story that explains a mathematical question, biases performance of low-SES students. In a large-scale cohort study of Trends in International Mathematics and Science Studies (TIMSS)—including data from 58 different countries from students in grades 4 and 8 (N = 5,501,165)—we examine whether item content that is more likely associated with major challenges for low-SES students (money, food, social relationships) improves their performance, compared with their average math performance. In contrast to our preregistered predictions, results show that low-SES students scored lower on items with this specific content than expected based on individual’s average performance. The effect sizes are considerable: on average, the chance to answer correctly is 18% lower. These results provide new insights in features of tests that disadvantage low-SES students and have implications for fair testing.