The distorting effects of deciding to stop sampling information

Abstract

This paper asks how strategies of information sampling are affected by a learner’s goal. Based on a theoretical analysis and two behavioral experiments, we show that learning goals have a crucial impact on the decision of when to stop sampling. This decision, in turn, affects the statistical properties (e.g. average values, or standard deviations) of the data collected under different goals. Specifically, we find that sampling with the goal of making a binary choice can introduce a correlation between the average value of a sample and its size (the number of values sampled). Across multiple rounds of sampling, this has the potential of biasing learn- ers’ inferences about the underlying process that generated the samples, specifically if learners ignore sample size when making these inferences. We find that people are indeed biased in this way and make different inferences about the same data-generating process when sampling with different learning goals. These findings highlight yet another danger of inferring general patterns from samples of evidence the learner had a hand in collecting.

Between heuristics and optimality: Flexible integration of cost and evidence during information sampling

“…Previous work has demonstrated that humans exhibit subjective biases in the computation and integration of accruing costs that limit the optimality of behavior (Cisek et al, 2009;Coenen & Gureckis, 2016;Furl & Averbeck, 2011;Hauser et al, 2017Hauser et al, , 2018. However, these policies still assume that humans engage in the computational expensive estimations of the value of stopping and the value of continuing in the same manner as an Ideal Observer (Bossaerts & Murawski, 2017;Payzan-LeNestour & Bossaerts, 2011).…”

“…During information sampling, individuals not only decide how to balance sampling costs with accuracy but also contend with balancing the costs and benefits of exerting control (Shenhav et al, 2013). Previous information sampling accounts have examined the impact of contexts such as changes in task difficulty (Coenen & Gureckis, 2016; Malhotra et al, 2017) and changes in sampling costs (Hauser et al, 2018; Juni et al, 2016) in altering sampling behavior but have not specifically examined if these contexts changed the underlying strategy. We examined the context of varying reward stakes on information sampling and found that while individuals maintained the same underlying strategy between both contexts, reward increased the overall information that people gathered.…”

“…Current normative models of sequential information sampling posit that an optimal information sampling strategy should compare the expected values of available actions (continue sampling, choose option A, or choose option B) before selecting the action with the highest expected value (Coenen & Gureckis, 2016;Furl & Averbeck, 2011;Hauser et al, 2017Hauser et al, , 2018Moutoussis et al, 2011). Computing these expected values requires several steps.…”

Rational inference strategies and the genesis of polarization and extremism

“…The key element of this decision process related to polarization is that the goal of a decision maker is not to gather a representative sample of information, but to tip the balance of information far enough in one direction to conclude A B or B A (where denotes a preference order or belief). The effect of adopting a balance-based goal was examined in work by (Coenen & Gureckis, 2016), where participants were presented with a deck of cards containing blue and red cards and asked to make a choice about whether there were more red or blue cards in the deck (choice condition) or estimate the proportion of red / blue cards in the deck (estimation condition). Critically, Coenen & Gureckis found that the distribution of samples gathered by participants in the choice condition was not representative of the actual proportion of red cards in the deck.…”

“…In this approach, a decision maker estimates the mean of the distribution when the error of their accumulated samples falls below a certain level. Some support for the efficacy of an estimation goal as an intervention to reduce the sampling issues in decision making is provided by Coenen & Gureckis (2016). Their studies contrasted choice against an estimation condition, where participants were asked to estimate the proportion of red or blue cards in a deck as opposed to simply deciding whether there were more red or blue cards.…”