How to ask twenty questions and win: Machine learning tools for assessing preferences from small samples of willingness-to-pay prices

Abstract

Subjective value has long been measured using binary choice experiments, yet responses like willingness-to-pay prices can be an effective and efficient way to assess individual differences risk preferences and value. Tony Marley's work illustrated that dynamic, stochastic models permit meaningful inferences about cognition from process-level data on paradigms beyond binary choice, yet many of these models remain difficult to use because their likelihoods must be approximated from simulation. In this paper, we develop and test an approach that uses deep neural networks to address this problem, performing instantaneous parameter estimation from a set of willingness-to-pay data. We show that a trained network is able to accurately recover true risk preferences related to utility, response caution, anchoring biases, and non-decision processes. This model was applied to estimate model parameters for a large, demographically representative sample of U.S. participants who completed a short, 20-question pricing task. It successfully characterizes the risk preferences of a diverse sample, showing that age and level of income are related to thresholds and anchoring biases, respectively. The results illustrate the utility of machine-learning approaches for wider adoption and integration of cognitive and economic models, providing efficient methods for quantifying meaningful differences in risk preferences from simple experiments.

Artificial neural networks for model identification and parameter estimation in computational cognitive models

“…particle filtering (Djuric et al, 2003); density estimation (Minka, 2013)); inverse binomial sampling (van Opheusden et al, 2020) may prove to be more robust, but frequently require more advanced mathematical knowledge and model case-based adaptations, or are more computationally expensive; indeed, some of them may not be usable or tractable in our type of data and models where there are sequential dependencies between trials Acerbi and Ma, 2017; van Opheusden et al, 2020. ANN-based methods such as ours or others’ Radev, Mertens, et al, 2020; Radev, Voss, et al, 2020; Sokratous et al, 2023, on the other hand, offers a more straightforward and time-efficient path to both parameter estimation and model identification. Developing more accessible and robust methods is critical for advances in computational modeling and cognitive science, and the rising popularity of deep learning puts neural networks forward as useful tools for this purpose.…”

“…These methods enable automated (or semi-automated) construction of summary statistics, minimizing the effect the choice of summary statistics may have on the accuracy of parameter estimation ( Y. Chen et al, 2020 ; Fearnhead and Prangle, 2012 ; Jiang et al, 2017 ; Lavin et al, 2021 ; Radev, Mertens, et al, 2020 ; Radev, Voss, et al, 2020 ). This innovative approach serves to amortize the computational cost of simulation-based inference, opening new frontiers in terms of scalability and performance ( Boelts et al, 2022 ; Fengler et al, 2021 ; Ghaderi-Kangavari et al, 2023 ; Radev, Mertens, et al, 2020 ; Radev, Voss, et al, 2020 ; Radev et al, 2021 ; Schmitt et al, 2021 ; Sokratous et al, 2023 ).…”

Improving the reliability and validity of the IAT with a dynamic model driven by similarity

“…While most solutions to this problem center around systemic issues like quantitative and computational training, it is also possible for modelers to make their models more accessible to a wide audience. We seek to accomplish this by using a new approach to modeling using neural networks (Radev, Mertens, Voss, Ardizzone, & Köthe, 2020;Radev et al, 2021;Lueckmann et al, 2019;Gutmann & Corander, 2016;Fengler et al, 2020;Cranmer et al, 2020;Sokratous et al, 2022). In this approach, rather than requiring a user to use a modeler's code to re-run their model on a new data set, a modeler instead trains a neural network to map input data (e.g., accuracy and response times) onto the most likely parameter estimates.…”

“…The neural network fitting approach we used Sokratous et al, 2022) is also in its infancy, having only been developed over the past few years. While it already shows impressive performance compared to traditional modeling methods -yielding equallyprecise estimates in a tiny fraction of the time -it is best used as a method for simulation-based models that are frequently applied to a common task.…”

Improving the reliability and validity of the IAT with a dynamic model driven by similarity

“…While most solutions to this problem center around systemic issues like quantitative and computational training, it is also possible for modelers to make their models more accessible to a wide audience. We seek to accomplish this by using a new approach to modeling using neural networks (Radev, Mertens, Voss, Ardizzone, & Köthe, 2020;Radev et al, 2021;Lueckmann et al, 2019;Gutmann & Corander, 2016;Fengler et al, 2020;Cranmer et al, 2020;Sokratous et al, 2022). In this approach, rather than requiring a user to use a modeler's code to re-run their model on a new data set, a modeler instead trains a neural network map input data (e.g., accuracy and response times) onto the most likely parameter estimates.…”