Psychometrics is not measurement: Unraveling a fundamental misconception in quantitative psychology and the complex network of its underlying fallacies.

Abstract

Psychometrics has always been confronted with fundamental criticism, highlighting serious insufficiencies and fallacies. Many fallacies persist, however, because each critic explores only some fallacies while still building on others. This article scrutinizes the epistemological, metatheoretical and methodological foundations of psychometrics, revealing a complex network of numerous conceptual fallacies underlying its framework of theory and practice. At its core lies a key challenge for psychology: the necessity to distinguish the phenomena under study from the means used to explore them (e.g., concepts, methods, data). This distinction is intricate because concepts constitute psychical phenomena in themselves and many psychical phenomena are accessible only through language-based methods. The analyses show how insufficient consideration of this important distinction and common misconceptions about concepts and language (e.g., signifier-referent conflation; reification of constructs) led to conflations of disparate notions of key terms in psychological measurement (e.g., 'variables', 'attributes', 'causality') and numerous interrelated fallacies (e.g., construct-referent conflation, phenomenon-quality-quantity conflation, numeral-number conflation). These fallacies are maintained and masked by repeated conceptual back-and-forth switching between two incompatible epistemological frameworks, 1) an operationist framework of data modelling implemented through methodical and statistical operations and 2) a realist framework of measurement sporadically invoked in theoretical considerations but neither theoretically elaborated nor empirically implemented. The analyses demonstrate that psychometrics constitutes only data modelling but not data generation or even measurement as often assumed and that analogies to (indirect or fundamental) physical measurement are mistaken. They provide theoretical support for the increasing criticism of psychometrics and its use in research and applied contexts.

Problematic research practices in psychology: Misconceptions about data collection entail serious fallacies in data analysis

“…Their frequent conflation entails numerous fallacies, such as when concepts describing the study phenomena are reified as real entities and erroneously equated with the phenomena underlying those described (e.g., "traits"), thus confusing description with explanation. Therefore, by using common jargon, even critics implicitly build on misconceptions that they rightfully criticise and that will persist unless psychologists establish a more elaborated research terminology (Uher, 2021b).…”

“…This applies in particular to measurement, which requires establishment of documented, unbroken measurand-result connections that allow tracing the results and their generation back to the properties and phenomena studied (Uher, 2018a(Uher, , 2020. But positivist beliefs and widespread misconceptions about sign systems lead many psychologists to ignore the inherently representational function of data, thereby conflating the study phenomena with the means used for their exploration (Uher, 2021b(Uher, , 2021c.…”

“…It also entails fallacies, such as the widespread assumption that between-individual and within-individual structures could be isomorphic, which ignores nonergodicity (Chirkov & Anderson, 2018;Molenaar & Campbell, 2009;Uher, 2015b). Instead, individual variability must be analysed, for which Arocha (2021) presents approaches from perceptual control theory and observation-oriented modelling-yet without specifying their relations to the Wundtian nomothetic methodology, which is needed to develop generalised knowledge about psychical processes and functioning from case-by-case based analyses (Salvatore & Valsiner, 2010;Uher, 2021b).…”

Functions of units, scales and quantitative data: Fundamental differences in numerical traceability between sciences

“…The demand for transparent and transferable quantitative information about human capital is growing (Fisher 2009)-as are discussions about replication crises (Hanfstingl 2019;Nosek et al 2015), validity (Buntins et al 2017;Newton 2012) and quantitative methods in psychology and social sciences (Michell 2003;Tafreshi et al 2016;Thomas 2020;Uher 2021cUher , 2021dValsiner 2017;Westerman 2014). Current debates primarily concern issues of data analysis (e.g., significance testing, effect sizes and robust statistics (Epskamp 2019;Open Science Collaboration 2015).…”

“…Much less attention is paid to the processes by which quantitative data are generated in the first place before they are being processed and analysed (Uher 2018a(Uher , b, c, 2019(Uher , 2021a. Indeed, many debates on 'measurement' (e.g., in psychometrics) actually concern only data modelling but not data generation (Uher 2021c(Uher , 2021d. Data analyses, however, can reveal valid information about the study objects only if-during data generation-relevant properties have been encoded into the data in appropriate ways.…”