Superforecasting reality check: Evidence from a small pool of experts and expedited identification

Katsagounos, Ilias; Thomakos, Dimitrios D.; Litsiou, Konstantia; Νικολόπουλος, Κωνσταντίνος

doi:10.1016/j.ejor.2020.06.042

Cited by 12 publications

(7 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Seekers often estimate that the main cost of an idea challenge comes from the reward itself; yet, making mistakes in the selection process for "building a crowd" (Dahlander & Piezunka, 2020) and picking the wrong idea might result in the seeker wasting time and money. Also, the scarcity of resources for engaging a vast number of external experts or judges and the effectiveness of a limited number of judges, has been questioned in the literature interested in crowdsourcing for judgmental and superforecasting (Katsagounos, Thomakos, Litsiou, & Nikolopoulos, 2021). Accordingly, we argue that the research contribution can be classified as "exaptation" (Gregor & Hevner, 2013), a known solution to a new problem.…”

Section: Introductionmentioning

confidence: 95%

Thinking about the seeker: a method for ideas selection in crowdsourcing through clustering

Viscusi¹,

Bonazzi

2021

Proceedings

View full text Add to dashboard Cite

This article presents a method for new ideas selection, which uses clustering to filter ideas while considering the seeker's goals and the learning dynamics. Most of the previous research has considered the idea provider as the main subject of analysis. Instead, we focus on the bounded rationality of the solution seeker. Seekers often estimate that the main cost of an idea challenge comes from the reward itself; yet, making mistakes in the selection process and picking the wrong idea might result in the seeker wasting time and money. The study discussed in the paper is an early-stage application of the design science research (DSR); thus, we argue that the research contribution can be classified as "exaptation", a known solution to a new problem.

show abstract

Section: Introductionmentioning

confidence: 95%

Thinking about the seeker: a method for ideas selection in crowdsourcing through clustering

Viscusi¹,

Bonazzi

2021

Proceedings

View full text Add to dashboard Cite

show abstract

“…There are probably correlates of decision tools' predictive validity that are observable 77,78,[110][111][112]79,93,98,[105][106][107][108][109] . There is a readable management literature 101,104 , reflecting a large technical literature 102,103,[113][114][115][116][117][118][119] , on how to tackle difficult evaluation tasks, and on how to decide if the benefit of evaluation is likely to outweigh the costs 101 (Figure 4). But this knowledge has been under-deployed, perhaps because the value of measurement is not clear unless one runs the decision theoretic maths 6,86 (Figure 3, Figure 4).…”

Section: [H2] Decision Tool Evaluationmentioning

confidence: 99%

Predictive validity in drug discovery: what it is, why it matters and how to improve it

Scannell

Bosley²,

Hickman³

et al. 2022

Nat Rev Drug Discov

View full text Add to dashboard Cite

Successful drug discovery is like finding oases of safety and efficacy in chemical and biological deserts. Screens in disease models, and other decision tools used in drug research and development (R&D), point towards oases when they score therapeutic candidates in a way that correlates with clinical utility in humans. Otherwise, they probably lead in the wrong direction. This line of thought can be quantified by using decision theory, in which 'predictive validity' is the correlation coefficient between the output of a decision tool and clinical utility across therapeutic candidates. Analyses based on this approach reveal that the detectability of good candidates is extremely sensitive to predictive validity, because the deserts are big and oases small. Both history and decision theory suggest that predictive validity is under-managed in drug R&D, not least because it is so hard to measure before projects succeed or fail later in the process. This article explains the influence of predictive validity on R&D productivity and discusses methods to evaluate and improve it, with the aim of supporting the application of more effective decision tools and catalysing investment in their creation.

show abstract

“…Subsequent research after the superforecasting experiment has included exploring further optimal forecasting tournament preparation (Horowtiz et al 2021;Katsagounos et al 2021) and extending Tetlock and Mellers' approach to answer broader, more time-distant questions (Page, Aiken, and Murdick 2020). It should be noted that there have been no recent advances on computational toolkits for the field similar to that proposed in this paper.…”

Section: Forecastingmentioning

confidence: 99%

Forecasting Argumentation Frameworks

Irwin¹,

Rago²,

Toni³

2022

Preprint

View full text Add to dashboard Cite

We introduce Forecasting Argumentation Frameworks (FAFs), a novel argumentation-based methodology for forecasting informed by recent judgmental forecasting research. FAFs comprise update frameworks which empower (human or artificial) agents to argue over time about the probability of outcomes, e.g. the winner of a political election or a fluctuation in inflation rates, whilst flagging perceived irrationality in the agents' behaviour with a view to improving their forecasting accuracy. FAFs include five argument types, amounting to standard pro/con arguments, as in bipolar argumentation, as well as novel proposal arguments and increase/decrease amendment arguments. We adapt an existing gradual semantics for bipolar argumentation to determine the aggregated dialectical strength of proposal arguments and define irrational behaviour. We then give a simple aggregation function which produces a final group forecast from rational agents' individual forecasts. We identify and study properties of FAFs and conduct an empirical evaluation which signals FAFs' potential to increase the forecasting accuracy of participants.

show abstract

Superforecasting reality check: Evidence from a small pool of experts and expedited identification

Cited by 12 publications

References 40 publications

Thinking about the seeker: a method for ideas selection in crowdsourcing through clustering

Thinking about the seeker: a method for ideas selection in crowdsourcing through clustering

Predictive validity in drug discovery: what it is, why it matters and how to improve it

Forecasting Argumentation Frameworks

Contact Info

Product

Resources

About