Quantitative Data From Rating Scales: An Epistemological and Methodological Enquiry

Abstract

Rating scales are popular methods for generating quantitative data directly by persons rather than automated technologies. But scholars increasingly challenge their foundations. This article contributes epistemological and methodological analyses of the processes involved in person-generated quantification. They are crucial for measurement because data analyses can reveal information about study phenomena only if relevant properties were encoded systematically in the data. The Transdisciplinary Philosophy-of-Science Paradigm for Research on Individuals (TPS-Paradigm) is applied to explore psychological and social-science concepts of measurement and quantification, including representational measurement theory, psychometric theories and their precursors in psychophysics. These are compared to theories from metrology specifying object-dependence of measurement processes and subject-independence of outcomes as key criteria, which allow tracing data to the instances measured and the ways they were quantified. Separate histories notwithstanding, the article’s basic premise is that general principles of scientific measurement and quantification should apply to all sciences. It elaborates principles by which these metrological criteria can be implemented also in psychology and social sciences, while considering their research objects’ peculiarities. Application of these principles is illustrated by quantifications of individual-specific behaviors (‘personality’). The demands rating methods impose on data-generating persons are deconstructed and compared with the demands involved in other quantitative methods (e.g., ethological observations). These analyses highlight problematic requirements for raters. Rating methods sufficiently specify neither the empirical study phenomena nor the symbolic systems used as data nor rules of assignment between them. Instead, pronounced individual differences in raters’ interpretation and use of items and scales indicate considerable subjectivity in data generation. Together with recoding scale categories into numbers, this introduces a twofold break in the traceability of rating data, compromising interpretability of findings. These insights question common reliability and validity concepts for ratings and provide novel explanations for replicability problems. Specifically, rating methods standardize only data formats but not the actual data generation. Measurement requires data generation processes to be adapted to the study phenomena’s properties and the measurement-executing persons’ abilities and interpretations, rather than to numerical outcome formats facilitating statistical analyses. Researchers must finally investigate how people actually generate ratings to specify the representational systems underlying rating data.

Measurement in metrology, psychology and social sciences: data generation traceability and numerical traceability as basic methodological principles applicable across sciences

“…Moreover, the TPS-Paradigm puts into focus the individuals who are doing the research and generating the data to help open up a meta-perspective on research processes, as done in this article. The TPS-Paradigm has already been applied (1) to integrate and expand on previous concepts of individuals' psyche, behaviour, language and contexts (Uher 2013;2015a, c;2016a, b); (2) to refine concepts and methodologies for comparing and taxonomising individual differences in various phenomena and populations (Uher 2015b, c, d, e;2018b, c), and (3) to critically analyse the involvement of human abilities in data generation across the empirical sciences (Uher 2019) as well as raters' use of standardised assessment scales (Uher 2018a). These conceptual developments and analyses are demonstrated in various empirical studies (e.g., Uher et al 2013a, b;Uher 2015d;Uher and Visalberghi 2016).…”

“…This term is very broad; for any specific case, it requires specification of what kind of information is converted in what ways into what other kind of information. This is commonly done explicitly in metrology, but not so in psychology and social sciences (Uher 2018a). In metrology, engineering and also in psychophysics, information conversion is commonly called transduction; in other fields, also translation or transcription (e.g., molecular biology).…”

“…That is, they must specify the decisions that the measurement-executing persons have to make about which pieces of information from the phenomena and properties under study should be demarcated and categorised in what ways, and the rules by which these should be assigned to the signifiers (e.g., mathematical symbols, lexical descriptions). These specifications must be made explicit and involve properties that are directly and accurately perceivable by data-generating persons during measurement execution (Uher 2018a(Uher , 2019.…”

“…This allows to trace the measurement results thus-created, in the inverse direction, back to the measurands and the particular references (e.g., standard units) used to quantify them (Fig. 2); this is called data-generation traceability in the TPS-Paradigm (Uher 2018a). Maybe this kind of traceability is so self-evident for metrologists and already implied by their concept of objectdependence, that it is not explicitly mentioned in metrological research, which focusses only on numerical traceability (see next).…”

“…Behaviours' pronounced variability, transient nature and context-dependent meaning entails, however, that a) the demarcation and categorisation of behavioural acts necessarily involves some defined scope for interpretation (e.g., what acts are of the same kind), and that b) observers, using nothing but their human abilities 16 , can hardly ever quantify particular properties directly (e.g., duration of talking). For these reasons, observers commonly encode only occurrences/non-occurrences of defined behavioural acts in nominal data (1/0) from which post hoc-after data generation is completed-or using behavioural coding software, ratioscaled quantitative data can be derived (e.g., durations, frequencies; for details (Uher 2013(Uher , 2015d(Uher , 2018a. Measurement of temporal properties is well-established; defined occurrences are discrete quantities (multitudes), which are countable.…”