Learning to allocate limited time to decisions with different expected outcomes

Abstract

The goal of this article is to investigate how human participants allocate their limited time to decisions with different properties. We report the results of two behavioral experiments. In each trial of the experiments, the participant must accumulate noisy information to make a decision. The participants received positive and negative rewards for their correct and incorrect decisions, respectively. The stimulus was designed such that decisions based on more accumulated information were more accurate but took longer. Therefore, the total outcome that a participant could achieve during the limited experiments' time depended on her "decision threshold", the amount of information she needed to make a decision. In the first experiment, two types of trials were intermixed randomly: hard and easy. Crucially, the hard trials were associated with smaller positive and negative rewards than the easy trials. A cue presented at the beginning of each trial would indicate the type of the upcoming trial. The optimal strategy was to adopt a small decision threshold for hard trials. The results showed that several of the participants did not learn this simple strategy. We then investigated how the participants adjusted their decision threshold based on the feedback they received in each trial. To this end, we developed and compared 10 computational models for adjusting the decision threshold. The models differ in their assumptions on the shape of the decision thresholds and the way the feedback is used to adjust the decision thresholds. The results of Bayesian model comparison showed that a model with time-varying thresholds whose parameters are updated by a reinforcement learning algorithm is the most likely model. In the second experiment, the cues were not presented. We showed that the optimal strategy is to use a single time-decreasing decision threshold for all trials. The results of the computational modeling showed that the participants did not use this optimal strategy. Instead, they attempted to detect the difficulty of the trial first and then set their decision threshold accordingly.

Mechanisms Underlying Decision-Making as Revealed by Deep-Brain Stimulation in Patients with Parkinson’s Disease

“…In a modeling framework, optimizing this trade-off between the speed and accuracy with which we make decisions can be implemented through a decision threshold that specifies the amount of evidence that is required for making a choice [1–3]. It has been inferred from behavioral data that humans may adjust their decision threshold on the basis of both the instruction to be fast or accurate given before a task [4, 5] and the difficulty of the decision to be made as the task unfolds [6–8]. The process of adjusting the decision threshold according to task difficulty was recently investigated in a behavioral study, which suggested that humans determine the difficulty of the current decision after a brief period of integrating evidence, and only then adjust the decision threshold in a single abrupt change [6].…”

The detour problem in a stochastic environment: Tolman revisited

“…One way to compare these models is to treat BIC/AIC as the log model evidence for each participant and investigate if there is a significant difference between the BIC (AIC) for a pair of models, Khodadadi et al (2017). However, this needs many pairwise comparisons.…”

A theoretical analysis of the reward rate optimality of collapsing decision criteria

“…Reinforcement learning models (Busemeyer and Stout, 2002;Sutton & Barton, 1998), for instance, have successfully been used to explain the acquisition of optimal decision policies in value-based decisionmaking (e.g., Ahn et al (2008), Fridberg et al (2010), and Steingroever et al (2014)). Such a combined model, as suggested by Khodadadi et al (2017), would allow researchers to account for factors such as incomplete exploration, and help quantify the degree of RR-optimality human decision-makers can achieve within a given time frame.…”