Optimal Temporal Risk Assessment

Abstract

Time is an essential feature of most decisions, because the reward earned from decisions frequently depends on the temporal statistics of the environment (e.g., on whether decisions must be made under deadlines). Accordingly, evolution appears to have favored a mechanism that predicts intervals in the seconds to minutes range with high accuracy on average, but significant variability from trial to trial. Importantly, the subjective sense of time that results is sufficiently imprecise that maximizing rewards in decision-making can require substantial behavioral adjustments (e.g., accumulating less evidence for a decision in order to beat a deadline). Reward maximization in many daily decisions therefore requires optimal temporal risk assessment. Here, we review the temporal decision-making literature, conduct secondary analyses of relevant published datasets, and analyze the results of a new experiment. The paper is organized in three parts. In the first part, we review literature and analyze existing data suggesting that animals take account of their inherent behavioral variability (their “endogenous timing uncertainty”) in temporal decision-making. In the second part, we review literature that quantitatively demonstrates nearly optimal temporal risk assessment with sub-second and supra-second intervals using perceptual tasks (with humans and mice) and motor timing tasks (with humans). We supplement this section with original research that tested human and rat performance on a task that requires finding the optimal balance between two time-dependent quantities for reward maximization. This optimal balance in turn depends on the level of timing uncertainty. Corroborating the reviewed literature, humans and rats exhibited nearly optimal temporal risk assessment in this task. In the third section, we discuss the role of timing uncertainty in reward maximization in two-choice perceptual decision-making tasks and review literature that implicates timing uncertainty as an important factor in performance quality. Together, these studies strongly support the hypothesis that animals take normative account of their endogenous timing uncertainty. By incorporating the psychophysics of interval timing into the study of reward maximization, our approach bridges empirical and theoretical gaps between the interval timing and decision-making literatures.

Mice can count and optimize count-based decisions

“…Our findings showed that animals can optimize the number of responses by integrating their numerosity-related uncertainty into their decisions and maximize the reward-rate attained in this task. These findings are consistent with the conclusions of previous studies that utilized temporal decision-making tasks; these studies showed that humans and other animals can adopt reward maximizing temporal decision strategies by taking account of their scalar endogenous timing uncertainty (Balcı et al, 2011;Çavdaroğlu et al, 2014). Thus, this study extended the scope of optimal temporal risk assessment to the domain of numerosity-based decision-making.…”

“…Reward-rate maximization in this task requires animals to aim at waiting longer than the minimum delay and how much more than the minimum delay animals should on average wait depends on their level of scalar endogenous timing uncertainty. Our previous findings showed that humans and non-human animals can optimize their wait time and maximize the reward-rate (Balcı et al, 2011;Çavdaroğlu et al, 2014; but see Freestone et al, 2015). The results of the current study showed that similar optimal decision-strategies can be attained when the reinforcement schedule is based on the number of responses rather than their timing.…”

“…There are now a number of studies showing that the decision process takes into account the uncertainty and its statistical properties when it sets the target (or decision threshold, for review see Balcı et al, 2011). Importantly, as summarized above these statistical properties of timing uncertainty are shared by numerical cognition pointing at a relationship between endogenous numerical uncertainty and reward maximizing numerical decisions.…”

Time-based reward maximization

“…Findings from earlier studies that adopted such an analytical approach [8][9][10][11][12] showed that both humans and rodents can adopt nearly optimal strategies by taking account of their endogenous uncertainty in tasks that required various decisions. In one of these experiments [8], humans and mice were trained in a temporal discrimination task that was composed of two types of trials (short latency trial and long latency trial) presented probabilistically.…”

Mice plan decision strategies based on previously learned time intervals, locations, and probabilities

“…The resultant timed behaviors have been shown to be sensitive to other crucial elements of environmental statistics, such as the probabilities of different outcomes (2)(3)(4). However, how steady-state choice behavior emerges in temporal decision-making tasks that contain probabilistic contingencies remains to be answered.…”