Sophisticated Statistics Cannot Compensate for Method Effects If Quantifiable Structure Is Compromised

Abstract

Researchers rely on psychometric principles when trying to gain understanding of unobservable psychological phenomena disconfounded from the methods used. Psychometric models provide us with tools to support this endeavour, but they are agnostic to the meaning researchers intend to attribute to the data. We define method effects as resulting from actions which weaken the psychometric structure of measurement, and argue that solution to this confounding will ultimately rest on testing whether data collected fit a psychometric model based on a substantive theory, rather than a search for a model that best fits the data. We highlight the importance of taking the notions of fundamental measurement seriously by reviewing distinctions between the Rasch measurement model and more generalised 2PL and 3PL IRT models. We then present two lines of research that highlight considerations of making method effects explicit in experimental designs. First, we contrast the use of experimental manipulations to study measurement reactivity during the assessment of metacognitive processes with factor-analytic research of the same. The former suggests differential performance-facilitating and -inhibiting reactivity as a function of other individual differences, whereas factor-analytic research suggests a ubiquitous monotonically predictive confidence factor. Second, we evaluate differential effects of context and source on within-individual variability indices of personality derived from multiple observations, highlighting again the importance of a structured and theoretically grounded observational framework. We conclude by arguing that substantive variables can act as method effects and should be considered at the time of design rather than after the fact, and without compromising measurement ideals.

Intelligence IS Cognitive Flexibility: Why Multilevel Models of Within-Individual Processes Are Needed to Realise This

“…Provision of feedback designed to change performance is one of the defining features of the dynamic testing paradigm ( Guthke and Beckmann 2000 ), but other forms of prompting may also change the way people approach problems. While the intention of such manipulations is to focus assessment on dynamic processes rather than static ones, an important theoretical implication of interpositions is that they may impact the validity of the assessment in unintended ways ( Birney et al 2022 ; Double and Birney 2019 ). Careful theorising and experimentation are necessary to ensure validity claims can be defended.…”

“…The same criticism would apply to an overuse of “shrinkage” in MLM regression models if this was observed to occur. Relying on sophisticated statistical tools to “purify” our measures from what are ultimately method-effects ( Birney et al 2022 ) reflects how little we understand about the sources of impurity (e.g., unreliability or multidimensionality) in our measures ( van der Maas et al 2017 ). We would do better to improve our measures using strong theory and better linked conceptual and empirical models, rather than make dubiously justified statistical adjustments.…”

“…This is because causal accounts put process and mechanism within the realm of direct observation through experimental manipulation and explicit process accounts, rather than leaving them to be inferred and reified from patterns of discriminant and convergent correlations after data have been collected. However, regardless of whether one studies formative or reflective concepts, or even network models, the burden of process identification is the responsibility of the researcher and cannot be delegated to statistics, no matter their levels of sophistication ( Birney et al 2022 ).…”

“…There is a subtle but important difference between a proposition that we should take dynamic processes seriously, and a claim that traditional psychometrics are not well suited to achieve this. We necessarily address these psychometric issues in Part 1 because they are, or at least should be, the pillars of operationalisation and measurement ( Birney et al 2022 ; Michell 1990 ).…”

Criterion-Referenced Assessment of Intelligence as Adaptation to the Environment: Is It Possible, Plausible, or Practical?

“…For example, Rasch-scaled measurement can provide scores that are norm-independent and that are invariant with respect to given item content (e.g., Stemler and Naples 2021 ). More generally, the field of assessment is moving toward theory-based assessments that could serve as bases of CRT and other innovations ( Birney et al 2022 ; Broers 2021 ; Kellen et al 2021 ). Whatever theory is used probably needs to consider that adaptive intelligence is multidimensional, at the very least with regard to the dimensions that underlie even narrowly defined intelligence ( Carroll 1993 ), creativity, and wisdom ( Sternberg and Glück 2019 ).…”