Representative Design in Psychological Assessment: A Case Study Using the Balloon Analogue Risk Task (BART)

Abstract

Representative design refers to the idea that experimental stimuli should be designed such that they represent the environments to which measured constructs are supposed to generalize. In this paper, we investigate the role of representative design in achieving valid and reliable psychological assessments, by focusing on a widely used behavioral measure of risk taking—the Balloon Analogue Risk Task (BART). We argue that the original implementation of the BART violates the principle of representative design, and by reanalyzing existing data demonstrate that people's expectations conflict with the task's stochastic structure—hence potentially hampering valid assessments. Moreover, we implemented two extensive studies (N = 772 and N = 632), aimed at experimentally testing the effects of improved representative designs in the BART. Although participants acquired more accurate beliefs about the optimal behavior due to these task adaptions, improving representative design proved to be no sufficient condition to boost the BART's psychometric properties (e.g., convergent validity, test–retest reliability). It follows that for the development of valid behavioral measurement instruments—as are needed, for instance, in neuroimaging studies—our field has to overcome the philosophy of the “repair program“ (i.e., fixing existing tasks). Instead, in order to develop valid tasks we suggest that ecological assessments are required to identify those real-life behaviors and associated psychological processes that lab-tasks are supposed to capture and generalize to.

Brain-outcome associations for risk taking depend on the measures used to capture individual differences

“…We still know comparatively little about how the brain integrates different aspects of a decision situation under uncertainty, especially when this is laden with incidental or contextual variables such as affect or prior experience (Knutson and Huettel, 2015;Samanez-Larkin and Knutson, 2015). To this end, it may be indispensable to develop novel and representative behavioral measures (Steiner and Frey, 2020) that facilitate the disentangling of risk-related cognitive and affective processes. Secondly, the assumption of generalizability from group to individuals is often not backed up by empirical evidence, posing a threat for individual studies and also research involving human subjects in general (Fisher et al, 2018).…”

Fear and anxiety in the face of COVID-19: Negative dispositions towards risk and uncertainty as vulnerability factors

“…Although some intraindividual variability exists across various domains of life, giving rise to domain-specific risk preferences (e.g., Weber, Blais, & Betz, 2002 ; Frey, Duncan, & Weber, 2020 ), an increasing amount of evidence suggests that people may also have relatively domain-general risk preferences (e.g., Frey et al, 2017 ; Highhouse, Nye, Zhang, & Rada, 2017 ; Zhang et al, 2019 ). Specifically, despite that various “risk-taking measures” are quite diverse and may vary in the extent to which they focus on risk as outcome variance (i.e., economic definition) versus risk as a threat (i.e., clinical definition), empirically there is a considerable convergence across measures – at least across different self-reports, but not necessarily across different behavioral tasks (for possible explanations for this observation, see Millroth, Juslin, Winman, Nilsson, & Lindskog, 2020 ; Steiner & Frey, 2021 ; Steiner, Seitz, & Frey, 2021 ).…”

Through the window of my mind: Mapping information integration and the cognitive representations underlying self-reported risk preference.

“…By contrast, their behavioral counterparts-that is, game-like tasks such as monetary lotteries-may be indispensable for applications such as examining the functional neural architecture of risk preference (e.g., Tisdall et al, 2018;Tom, Fox, Trepel, & Poldrack, 2007), but generally tend to be more intricate to implement (cf. Andreoni & Kuhn, 2019;Pedroni et al, 2017), and often fail to meet fundamental measurement properties (i.e., reliability, predictive validity; Beauchamp et al, 2017;Berg, Dickhaut, & McCabe, 2005;Eisenberg et al, 2019;Frey et al, 2017;Lönnqvist et al, 2015;Mata et al, 2018;Steiner & Frey, 2020).…”