Calibrating the experimental measurement of psychological attributes

Abstract

Experimental psychologists often seek to measure latent psychological attributes. A plethora of observation and data transformation methods are being used simultaneously in any given field, with few criteria to arbitrate between them. In this theoretical note, we extend classical validity theory by suggesting to use intended values in a benchmark experiment as singular validity criterion, which we term retrodictive validity. We formally introduce a statistical model of this situation. We mathematically derive that maximisation of retrodictive validity is guaranteed to maximise measurement accuracy as long as experimental aberration and measurement error are uncorrelated. In this case, retrodictive validity provides a lower bound on measurement accuracy. If this correlation is non-zero but positive, or negative and small, then maximisation of retrodictive validity has a potential, but is not guaranteed, to maximise measurement accuracy. In contrast, the method makes no assumption on the absolute size of the experimental aberration. This implies that weak experimental manipulations or inaccurate theoretical predictions can be leveraged for assessing measurement accuracy. We suggest that the criterion can be used for rational selection between measurement methods, including the choice of observables and of transformation methods or measurement models.

Computational modeling of threat learning reveals links with anxiety and neuroanatomy in humans

“…Third, only structural MRI data were examined; functional imaging during threat learning could reveal additional correlates of learning circuitry ( Homan et al, 2019 ), although the delivery of aversive stimuli to participants with anxiety in the MRI scanner presents a challenge ( Thorpe et al, 2008 ). Fourth, SCR data were analyzed using a single method, which, while established, relies only on directly-observable effects; future studies may consider using novel, computational analysis methods which could potentially reveal effects not observed using the current method ( Bach et al, 2018 ; Bach and Friston, 2013 ; Bach et al, 2020 ; Ojala and Bach, 2020 ). Along these lines, a multiverse approach may be used in future work to comprehensively compare multiple methods of quantifying threat learning.…”

Evidence for a minimal role of stimulus awareness in reversal of threat learning

“…Compared with peak-scoring methods for SCR, this approach has been shown to better discriminate between responses to aversive and neutral stimuli ( Bach et al 2009 ; Bach and Friston 2013 ) and to better discriminate between responses to CS+ and CS− in fear conditioning ( Bach et al 2010 ). This better discrimination implies better accuracy and precision than standard SCR analysis in inferences on the latent CS–US association ( Bach et al 2020 ).…”

Cross-species anxiety tests in psychiatry: pitfalls and promises

“…Establishing the latter may require a clearer view on the costs of pre-clinical trials, which are dictated by achievable effect sizes. These can be computed from behavioural variability in (untreated) control groups [ 127 , 128 ], and define statistical power for a human screening trial. Some promising steps into this direction have been taken for the NPU task [ 129 ].…”