Probabilistic forecasts have good 'external correspondence' if events that are assigned probabilities close to I tend to occur frequently, whereas those assigned probabilities near 0 tend to occur rarely. This paper describes simple procedures for analysing external correspondence into meaningful components that might guide efforts to understand and improve forecasting performance. The procedures focus on differences between the judgements made by the forecaster when the target event occurs, as compared to when it does not. The illustrations involve a professional oddsmaker's predictions of baseball game outcomes, meteorologists' precipitation forecasts and physicians' diagnoses of pneumonia. The illustrations demonstrate the ability of the procedures to highlight important forecasting tendencies that are sometimes more difficult to discern by other means.
KEY WORDS External correspondence Scoring rule Probabilisticforecasts Subjective probability Most forecasts are uncertain. That is, the forecaster is not absolutely sure that the events in question will occur. A good case can be made that such uncertainty should be made explicit and quantified. Then, as is done in decision analysis and related procedures (Malone, 1957; Raiffa, 1968; SRI Decision Analysis Group, 1977; Thompson and Brier, 1955; Winkler, 1972), trade-offs can be made between the values of various outcomes and the chances of their occurrence. Probabilistic forecasts are said to exhibit good 'external correspondence' when events that are assigned probabilities close to 1 occur frequently, whereas those assigned probabilities near 0 occur rarely (Yates, 1982). The concept of external forecast correspondence is closely related to the more general notion of 'accuracy ' (Daan and Murphy, 1982), or "'validity", i.e. the association between the probability statements and the actual outcomes ' (Murphy and Winkler, 1970, p. 281). Clearly, every forecaster aims for good external correspondence; the better the external correspondence, the more likely it is that decisions predicated on the given forecasts will turn out well.External correspondence is not a unitary construct. The characteristic quality of one forecaster's judgements can differ from that of another's in several respects. Thus, if one wanted to improve the