Verification of Forecasts Expressed in Terms of Probability

Brier, Glenn W.

doi:10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2

Cited by 4,154 publications

(2,696 citation statements)

References 0 publications

Supporting

Mentioning

2,672

Contrasting

Unclassified

Order By: Relevance

“…In particular, the MSD score underlies traditional statistical methods (like regression and the t-test), is a proper scoring rule (Brier, 1950;Selten, 1998), and can be translated to the intuitive Equivalent Number of Observation (ENO) score explained below.…”

Section: Competition Criterion: Mean Squared Deviation (Msd)mentioning

confidence: 99%

From anomalies to forecasts: Toward a descriptive model of decisions under risk, under ambiguity, and from experience.

Erev¹,

Ert

Plonsky³

et al. 2017

Psychological Review

210

258

View full text Add to dashboard Cite

Experimental studies of choice behavior document distinct, and sometimes contradictory, deviations from maximization. For example, people tend to overweight rare events in oneshot decisions under risk, and to exhibit the opposite bias when they rely on past experience.The common explanations of these results assume that the contradicting anomalies reflect situation-specific processes that involve the weighting of subjective values and the use of simple heuristics. The current paper analyzes 14 choice anomalies that have been described by different models, including the Allais, St. Petersburg, and Ellsberg paradoxes, and the reflection effect. Next, it uses a choice prediction competition methodology to clarify the interaction between the different anomalies. It focuses on decisions under risk (known payoff distributions) and under ambiguity (unknown probabilities), with and without feedback concerning the outcomes of past choices. The results demonstrate that it is not necessary to assume situation-specific processes. The distinct anomalies can be captured by assuming high sensitivity to the expected return and four additional tendencies: pessimism, bias toward equal weighting, sensitivity to payoff sign, and an effort to minimize the probability of immediate regret. Importantly, feedback increases sensitivity to probability of regret. Simple abstractions of these assumptions, variants of the model Best Estimate And Sampling Tools (BEAST), allow surprisingly accurate ex ante predictions of behavior. Unlike the popular models, BEAST does not assume subjective weighting functions or cognitive shortcuts.Rather, it assumes the use of sampling tools and reliance on small samples, in addition to the estimation of the expected values.

show abstract

Section: Competition Criterion: Mean Squared Deviation (Msd)mentioning

confidence: 99%

From anomalies to forecasts: Toward a descriptive model of decisions under risk, under ambiguity, and from experience.

Erev¹,

Ert

Plonsky³

et al. 2017

Psychological Review

210

258

View full text Add to dashboard Cite

show abstract

“…The values of the performance measures were then averaged across all 1000 datasets. For each scenario separately, the apparent performance of the different methods to handle treatment in the development of a prognostic model was compared using the Brier score, 13 Harrell's c-statistic, 14 the observed by expected risk prediction ratio, the standard errors of the association between the predictor X 1 and the outcome, and the proportion of incorrect treatment decisions. These performance measures were also compared against their optimal value, which was calculated based on the data generating model (model 2 in Section 3.2).…”

Section: Step 3: Calculate and Compare Parameters Of Apparent Performmentioning

confidence: 99%

Explicit inclusion of treatment in prognostic modeling was recommended in observational and randomized settings

Groenwold

Moons

Pajouheshnia

et al. 2016

Journal of Clinical Epidemiology

View full text Add to dashboard Cite

What is new• When developing a prognostic model (that aims to produce accurate probabilities of the outcome in case the patient is not treated) using data from a randomised trial in which individuals from one arm do not receive treatment, restricting the analysis to untreated individuals may be a suitable strategy. However, removing all patients in the treatment group will reduce the sample size, leading to greater uncertainty around predictions and also to prognostic models that are more prone to overfitting.• When developing a prognostic model using data from observational studies with treated patients, restricting the analysis to untreated individuals is not appropriate if treatment status depends on patient characteristics, including the predictors of the developed model.• For either randomised or observational studies, it is preferable to explicitly model treatment when developing a prognostic model.

show abstract

“…A scoring rule is called proper (strictly proper) if it is minimized (uniquely) at the true status probability Fðt j X i Þ (Savage, 1971;Gneiting and Raftery, 2007). An often considered strictly proper scoring rule is the Brier score (Brier, 1950). It is given by the squared distance between the patients observed status and the predicted probability, viz.…”

Section: Expected Lossmentioning

confidence: 99%

The Performance of Risk Prediction Models

Gerds¹,

Cai²,

2008

View full text Add to dashboard Cite

SummaryFor medical decision making and patient information, predictions of future status variables play an important role. Risk prediction models can be derived with many different statistical approaches. To compare them, measures of predictive performance are derived from ROC methodology and from probability forecasting theory. These tools can be applied to assess single markers, multivariable regression models and complex model selection algorithms. This article provides a systematic review of the modern way of assessing risk prediction models. Particular attention is put on proper benchmarks and resampling techniques that are important for the interpretation of measured performance. All methods are illustrated with data from a clinical study in head and neck cancer patients.

show abstract

Verification of Forecasts Expressed in Terms of Probability

Cited by 4,154 publications

References 0 publications

From anomalies to forecasts: Toward a descriptive model of decisions under risk, under ambiguity, and from experience.

From anomalies to forecasts: Toward a descriptive model of decisions under risk, under ambiguity, and from experience.

Explicit inclusion of treatment in prognostic modeling was recommended in observational and randomized settings

The Performance of Risk Prediction Models

Contact Info

Product

Resources

About