Rapid, precise, and reliable measurement of delay discounting using a Bayesian learning algorithm

Ahn, Woo-Young; Gu, Hairong; Shen, Yitong; Haines, Nathaniel; Hahn, Hunter; Teater, Julie; Myung, Jay I.; Pitt, Mark A.

doi:10.1101/567412

Cited by 14 publications

(22 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second, we analyzed data from Gawronski et al (2017), who collected data on the Self-Concept (introversion/extraversion) and Race (Black/White) versions of the Implicit Association Test (IAT). Lastly, we analyzed data from Ahn et al (2020), who collected data on the delay discounting task. Individually, each of these behavioral tasks has produced a deep body of literature-the Stroop, Flanker, and Posner Cueing tasks have been used extensively to develop theories of attention and inhibitory control, the IAT has been used to develop theories of implicit cognition and evaluations, and the delay discounting task has been used to develop theories of impulsivity and self-control.…”

Section: Datasets and Behavioral Paradigmsmentioning

confidence: 99%

“…In the context of the generative models developed above, the objective of ADO is to optimize parameter estimation, which can substantially improve reliability. For example, in delay discounting tasks, ADO can identify combinations of rewards and delays that optimize estimation of discounting rates, achieving test-retest reliabilities greater than r = .95 in fewer than 20 trials (Ahn et al, 2020). ADO is currently underutilized in most areas of behavioral science, but user-friendly software packages are now available (Yang et al, in press).…”

Section: Future Directionsmentioning

confidence: 99%

See 1 more Smart Citation

Theoretically Informed Generative Models Can Advance the Psychological and Brain Sciences: Lessons from the Reliability Paradox

Haines¹,

Pd²,

Lh³

et al. 2020

Preprint

Self Cite

106

119

View full text Add to dashboard Cite

Behavioral tasks (e.g., Stroop task) that produce replicable group-level effects (e.g., Stroop effect) often fail to reliably capture individual differences between participants (e.g., low test-retest reliability). This “reliability paradox” has led many researchers to conclude that most behavioral tasks cannot be used to develop and advance theories of individual differences. However, these conclusions are derived from statistical models that provide only superficial summary descriptions of behavioral data, thereby ignoring theoretically-relevant data-generating mechanisms that underly individual-level behavior. More generally, such descriptive methods lack the flexibility to test and develop increasingly complex theories of individual differences. To resolve this theory-description gap, we present generative modeling approaches, which involve using background knowledge to specify how behavior is generated at the individual level, and in turn how the distributions of individual-level mechanisms are characterized at the group level—all in a single joint model. Generative modeling shifts our focus away from estimating descriptive statistical “effects” toward estimating psychologically meaningful parameters, while simultaneously accounting for measurement error that would otherwise attenuate individual difference correlations. Using simulations and empirical data from the Implicit Association Test and Stroop, Flanker, Posner Cueing, and Delay Discounting tasks, we demonstrate how generative models yield (1) higher test-retest reliability estimates, and (2) more theoretically informative parameter estimates relative to traditional statistical approaches. Our results reclaim optimism regarding the utility of behavioral paradigms for testing and advancing theories of individual differences, and emphasize the importance of formally specifying and checking model assumptions to reduce theory-description gaps and facilitate principled theory development.

show abstract

Section: Datasets and Behavioral Paradigmsmentioning

confidence: 99%

Section: Future Directionsmentioning

confidence: 99%

Theoretically Informed Generative Models Can Advance the Psychological and Brain Sciences: Lessons from the Reliability Paradox

Haines¹,

Pd²,

Lh³

et al. 2020

Preprint

Self Cite

106

119

View full text Add to dashboard Cite

show abstract

“…Employing insights of computational psychiatry in a clinical setting will require the development of behavioral or physiological diagnostic tests, based on procedures like model selection and parameter estimation [59]. This presents a problem analogous to the design of experiments [60], and therefore computational optimization may be similarly useful in designing accurate and efficient diagnostic tests in this domain.…”

Section: Resultsmentioning

confidence: 99%

Computational optimization of associative learning experiments

Melinščak

Bach

2020

PLoS Comput Biol

View full text Add to dashboard Cite

With computational biology striving to provide more accurate theoretical accounts of biological systems, use of increasingly complex computational models seems inevitable. However, this trend engenders a challenge of optimal experimental design: due to the flexibility of complex models, it is difficult to intuitively design experiments that will efficiently expose differences between candidate models or allow accurate estimation of their parameters. This challenge is well exemplified in associative learning research. Associative learning theory has a rich tradition of computational modeling, resulting in a growing space of increasingly complex models, which in turn renders manual design of informative experiments difficult. Here we propose a novel method for computational optimization of associative learning experiments. We first formalize associative learning experiments using a low number of tunable design variables, to make optimization tractable. Next, we combine simulation-based Bayesian experimental design with Bayesian optimization to arrive at a flexible method of tuning design variables. Finally, we validate the proposed method through extensive simulations covering both the objectives of accurate parameter estimation and model selection. The validation results show that computationally optimized experimental designs have the potential to substantially improve upon manual designs drawn from the literature, even when prior information guiding the optimization is scarce. Computational optimization of experiments may help address recent concerns over reproducibility by increasing the expected utility of studies, and it may even incentivize practices such as study pre-registration, since optimization requires a pre-specified analysis plan. Moreover, design optimization has the potential not only to improve basic research in domains such as associative learning, but also to play an important role in translational research. For example, design of behavioral and physiological diagnostic tests in the nascent field of computational psychiatry could benefit from an optimization-based approach, similar to the one presented here.

show abstract

“…The use of more specific models will affect the BFDA such that the same data will become more diagnostic: a fixed sample size is projected to yield more evidence, and a given level of evidence is reached with smaller samples. Similarly, in Adaptive Design Optimization (ADO; e.g., Ahn et al, 2020;Myung et al, 2013) the next stimulus to be presented is determined by maximizing expected information gain. As the observations accumulate, the rival hypotheses become increasingly specific (i.e., their constituent posterior distributions become more peaked), and therefore easier to discriminate.…”

Section: Advantages Of Specific Hypothesis Testingmentioning

confidence: 99%

A Bayesian Perspective on Severity: Risky Predictions and Specific Hypotheses

Nnn¹,

Wagenmakers²,

Sprenger³

2020

Preprint

View full text Add to dashboard Cite

A tradition that goes back to Karl R. Popper assesses the value of a statistical test primarily by its severity: was it a honest and stringent attempt to prove the theory wrong? For "error statisticians" such as Deborah Mayo (1996, 2018), and frequentists more generally, severity is a key virtue in hypothesis tests. Conversely, failure to incorporate severity into statistical inference, as it allegedly happens in Bayesian inference, counts as a major methodological shortcoming. Our paper pursues a double goal: First, we argue that the error-statistical explication of severity has substantive drawbacks (i.e., neglect of research context; lack of connection to specificity of predictions; problematic similarity of degrees of severity to one-sided p-values). Second, we argue that severity matters for Bayesian inference via the value of specific, risky predictions: severity boosts the expected evidential value of a Bayesian hypothesis test. We illustrate severity-based reasoning in Bayesian statistics by means of a practical example and discuss its advantages and potential drawbacks.

show abstract

Rapid, precise, and reliable measurement of delay discounting using a Bayesian learning algorithm

Cited by 14 publications

References 54 publications

Theoretically Informed Generative Models Can Advance the Psychological and Brain Sciences: Lessons from the Reliability Paradox

Theoretically Informed Generative Models Can Advance the Psychological and Brain Sciences: Lessons from the Reliability Paradox

Computational optimization of associative learning experiments

A Bayesian Perspective on Severity: Risky Predictions and Specific Hypotheses

Contact Info

Product

Resources

About