Reducing the Bias of Norm Scores in Non-Representative Samples: Weighting as an Adjunct to Continuous Norming Methods

Abstract

We investigated whether the accuracy of normed test scores derived from non-demographically representative samples can be improved by combining continuous norming methods with compensatory weighting of test results. To this end, we introduce Raking, a method from social sciences, to psychometrics. In a simulated reference population, we modeled a latent cognitive ability with a typical developmental gradient, along with three demographic variables that were correlated to varying degrees with the latent ability. We simulated five additional populations representing patterns of non-representativeness that might be encountered in the real world. We subsequently drew smaller normative samples from each population and used an one-parameter logistic Item Response Theory (IRT) model to generate simulated test results for each individual. Using these simulated data, we applied norming techniques, both with and without compensatory weighting. Weighting reduced the bias of the norm scores when the degree of non-representativeness was moderate, with only a small risk of generating new biases.

Patients’ experiences of continuity of care: What is needed and how can it be measured?

“…Selection bias cannot be excluded as studies III and IV did not allow any drop-out analysis, given that not all HCs followed the instruction to register all patients who were informed about the study and asked to participate. In psychometric studies, representative samples are of great importance for developing norm scores (Gary et al, 2023), but are not as important for other types of psychometric evaluations that are relate more to the relevance of the sample (i.e., that the sample matches the intention of the measure), variations in the concept of the measure, and variations in the sample. Regarding this, the sample showed good variation in age, sex, self-reported diseases, and education.…”