From Noise to Bias: Overconfidence in New Product Forecasting

Abstract

We study decision behavior in the selection, forecasting, and production for a new product. In a stylized behavioral model and five experiments, we generate new insight into when and why this combination of tasks can lead to overconfidence (specifically, overestimating the demand). We theorize that cognitive limitations lead to noisy interpretations of signal information, which itself is noisy. Because people are statistically naive, they directly use their noisy interpretation of the signal information as their forecast, thereby underaccounting for the uncertainty that underlies it. This process leads to unbiased forecast errors when considering products in isolation, but leads to positively biased forecasts for the products people choose to launch due to a selection effect. We show that this selection-driven overconfidence can be sufficiently problematic that, under certain conditions, choosing the product randomly can actually yield higher profits than when individuals themselves choose the product to launch. We provide mechanism evidence by manipulating the interpretation noise through information complexity—showing that even when the information is equivalent from a Bayesian perspective, more complicated information leads to more noise, which, in turn, leads to more overconfidence in the chosen products. Finally, we leverage this insight to show that getting a second independent forecast for a chosen product can significantly mitigate the overconfidence problem, even when both individuals have the same information. This paper was accepted by Charles Corbett, operations management.

Can individual subjective confidence in training questions predict group performance in test questions?

“…Previous studies showed that simply aggregating two or more people's judgments by, for example, using a majority rule or averaging numerical estimations (i.e., group judgments) can often be more accurate than one person's judgment. This effect, known as the wisdom of crowds effect [1][2][3][4][5][6][7][8], has been observed in various fields including general knowledge tasks [9,10], forecasts [11,12], and matter of taste [13].…”

Explainability's Gain is Optimality's Loss? -- How Explanations Bias Decision-making

“…In marketing, targeting customers with the highest churn probability is found not to be the optimal decision [Ascarza 2018;Lemmens and Gupta 2020]. [Feiler and Tong 2021] shows that overconfidence in product forecast can lead to bad decisions and profit loss. In our study we demonstrate empirically that, whatever the accuracy and efficiency of a machine learning algorithm in evaluating alternatives with respect to some metric, explanations can introduce human biases and make the decision-maker more confident about some alternatives relative to others.…”

Can individual subjective confidence in prior questions predict group performance in future questions?

“…In contrast, if individuals with higher confidence and those with lower (not so high) confidence in prior question(s) are aggregated, then the group may likely be able to avoid decreasing accuracy in future questions. As described earlier, the diversity of judgments in a group is important in achieving the wisdom of crowds because individuals' judgmental errors are sometimes canceled out by collecting two or more diverse judgments [1,12,14,[34][35][36]. Because some individuals with high confidence may be overconfident or could solve prior question(s) by chance, their errors are likely to be canceled out by mixing individuals with lower confidence into a group (individuals with low confidence may have lack of knowledge and therefore sometimes make random guesses.…”

“…In such situations, aggregating both individuals with high confidence and those with middlelevel (not very high but not very low) confidence will be a good strategy. Previous studies on the wisdom of crowds highlighted the importance of a diversity of judgments in a group [1,3,12,34,48] because, especially in a binary choice question, various judgments are likely to cancel out individuals' judgmental biases by the majority rule. As shown in our analyses of accordance rates, individuals with high confidence tended to make similar judgments, whereas those with lower confidence tended to make more diverse judgments (probably because they would have less knowledge and would sometimes make random guesses).…”

“…Previous studies showed that simply aggregating two or more people's judgments by, for example, using a majority rule or averaging numerical estimations (i.e., group judgments) can often be more accurate than one person's judgment. This effect is known as the wisdom of crowds effect [1][2][3][4][5][6][7][8], and has been observed in various fields including general knowledge tasks [9,10], forecasts [11,12], and matter of taste [13].…”