Beating the average forecast: Regularization based on forecaster attributes

Abstract

In a variety of real-world forecasting contexts, researchers have demonstrated that the unweighted average forecast is reasonably accurate and difficult to improve upon with more complex, model-based aggregation methods. We investigate this phenomenon by systematically examining the relationship between individual forecaster characteristics (e.g., bias, consistency) and aspects of the criterion being forecast (e.g., "signal strength"). To this end, we develop a model inspired by Cultural Consensus Theory (Batchelder & Romney, 1988) that (i) allows us to jointly estimate both forecaster characteristics and environmental characteristics and (ii) contains the unweighted average as a special case. This allows us to use the model as a regularization method for forecast aggregation, where restrictions on forecaster parameters make the model similar to use of an unweighted average. Relatedly, the model allows us to apply existing results on optimal forecaster weighting to real data. We show how the model provides guidance for identifying prediction environments where the average forecast can potentially be beaten. We also conduct two simulation studies and illustrate the model's practical application using forecasts of Australian Football League point spreads. BackgroundA considerable body of work in forecasting has established that taking a simple, unweighted average of forecasters yields a forecast that is not only good, it is extremely difficult to improve upon with more complex aggregation methods (

Sequential Collaboration: The Accuracy of Dependent, Incremental Judgments

“…Assuming that experts are more likely to revise a judgment whereas novices more often use of the opportunity to opt out of answering, sequential collaboration results in an implicit weighting of expertise. This may explain the high accuracy of sequential collaboration, given that the weighting of expertise has been shown to be beneficial for improving wisdom-of-crowds estimates (Budescu & Chen, 2014;Merkle et al, 2020).…”

“…Recently, research has focused on methods to further improve the accuracy of wisdom of crowds by considering information about the group members (Budescu & Chen, 2014; e.g., Merkle et al, 2020). These extensions are based on the idea that group estimates can be even more accurate if individual judgments are not weighted equally but are rather weighted based on the expertise of the judges.…”

“…They found that weighing individual judgments by this measure of contribution improved the wisdom-of-crowds estimate. Based on a similar idea, Merkle et al (2020) developed a statistical model that optimally weighs judgments based on forecaster attributes thereby outperforming the unweighted aggregation of individual judgments. Overall, these examples show that taking into account individuals' expertise improves the accuracy of aggregated group estimates and that equal weighting ignores the potential benefits gained from assigning larger weights to experts.…”

“…Hence, maintaining judgments by opting out could be SEQUENTIAL COLLABORATION 14 considered as an implicit weighing by expertise in which contributors with lower expertise give more weight to judgments of contributors with higher expertise. Given that weighting by expertise is known to increase accuracy (Budescu & Chen, 2014;Merkle et al, 2020), we expect that this will lead to increasingly accurate judgments throughout a sequential chain.…”

The benefit of collaboration: Disentangling the sources of synergy in group judgments

“…There is a vast literature showing the benefit of averaging (e.g., see for an overview Mannes, Larrick, & Soll, 2012). Averaging is quite often the most accurate solution in many forecasting tasks (Merkle, Saw, & Davis-Stober, 2020). However, averaging does not represent a real group effect , because it does not require a group to form the average of independent individual judgments.…”