Choosing the Devil You Don’t Know: Evidence for Limited Sensitivity to Sample Size–Based Uncertainty When It Offers an Advantage

Abstract

Abstract. Many decision makers seek to optimize choices between uncertain options such as strategies, employees, or products. When performance targets must be met, attending to observed past performance is not enough to optimize choices-option uncertainty must also be considered. For example, for stretch targets that exceed observed performance, more uncertain options are often better bets. A significant determinant of option uncertainty is sample size: for a given option, the smaller the sample of information we have about it, the greater the uncertainty. In two studies, choices were made between pairs of uncertain options with the goal of exceeding a specified performance target. Information about the options differed in the size of the sample drawn from them, sample size, and the observed performance of those samples, the proportion of successes or "hits" in the sample. We found people to be sensitive to sample size-based uncertainty only when differences in observed performance were negligible. We conclude that in the presence of performance targets, people largely fail to capitalize on the value advantages of small samples in the presence of stretch targets.History: Accepted by Yuval Rottenstreich, judgment and decision making.

The influence of sample size on preferences from experience

“…Surprisingly, participants expressed higher confidence in their judgements when the number of coin tosses was small, whereas from a statistical perspective, confidence should be lower for smaller samples. Several studies have found that people’s inferential (e.g., Kutzner et al, 2017), perceptual (e.g., Kvam & Pleskac, 2016), and preferential judgements (e.g., Powell et al, 2017) are less sensitive to sample sizes than expected by normative models.…”

Relative insensitivity to sample sizes in judgments of frequency distributions: People are similarly confident in the results from 30 versus 3,000 observations.

“…For instance, participants estimated that a large hospital with 45 births daily and a small hospital with 15 births daily would have a similar number of days on which more than 60% of births were boys, whereas in reality, the larger hospital is much less likely to deviate from the mean of 50% than the smaller hospital (Kahneman & Tversky, 1972). People fail to realize that the means from smaller samples are likely to be more variable than means from larger samples drawn from the same population, indicating that people are largely insensitive to variations in the sample size when making judgments about variability in the sample mean (Kahneman & Tversky, 1972; Kutzner et al, 2016). Research on decision-making based on experiences has also found that people have higher confidence in data from single, isolated observations or anecdotes than data from repeated observations summarized in the form of statistical evidence (Obrecht et al, 2007), indicating that people do not take the sample size into consideration when interpreting data.…”