Assessing Bias and Accuracy in the World Bank-IMF's Debt Sustainability Framework for Low-Income Countries

Abstract

The World Bank and the IMF have adopted a debt sustainability framework (DSF) to evaluate the risk of debt distress in Low Income Countries (LICs). At the core of the DSF are empirically-based thresholds for each of five different measures of the debt burden (the "debt threshold approach" DTA). The DSF contains a rule for aggregating the information contained in these five different variables which we label the "worst-case aggregator" (WCA) in view of the fact that the DSF considers a breach of any one of the thresholds sufficient to indicate a high risk of debt distress. However, neither the DTA nor the WCA has heretofore been subject to empirical testing. We find that: (1) the DTA loses information relative to a simple proposed alternative; (2) the WCA is too conservative (predicting crises too often) in terms of the loss function used in the DSF; and (3) the WCA is less accurate than some simple proposed alternative aggregators as a predictor of debt distress.

Debt Sustainability in Low-Income Countries: Policies, Institutions, or Shocks?

“…Recently Berg et al (2014) examined how the DSF uses and aggregates the information contained in five separate debt indicators, taking as given the KN-based probit regressions in the current DSF. They found that using debt thresholds, rather than probability thresholds, results in a loss of country-specific information that reduces accuracy of predictions.…”

Predicting Fiscal Crises: A Machine Learning Approach

“…This approach relies on the Generalized Linear Model (GLM -typically the probit or logit version). Several studies use GLM to identify the determinants of fiscal crises, including Marashaden (1997), Detragiache and Spilimbergo (2001), Peter (2002), Manasse et al (2003), Ciarlone and Trebeschi (2005), Kraay and Nehru (2006), Gourinchas and Obstfeld (2012), Berg et al (2014), Dawood, Horsewood, andStrobel (2017), andPamies Sumner andBerti (2017). 10 Probit models also have an important role in policy practice, notably in the IMF/World Bank debt sustainability framework for low-income countries (see IMF, 2015).…”

“…I also display receiver-operator characteristic curves which, for each possible probability cutoff, plot true positives against false positives. And I report the area under the receiveroperator characteristics curve (AUROC), a popular measure of how well the rankings of probabilities correspond to observed outcomes (see, e.g., Jorda et al (2011) or Berg et al (2014)). Intuitively, the AUROC captures the probability that an algorithm selects the crisis out of a randomly selected crisis and non-crisis case.…”

“…That is, starting with the intercept, I expand the model by iteratively adding the variable that maximizes the model fit, evaluated using the Bayesian Information Criterion (BIC), until there is no more scope for improvement. The BIC offers a parametric approach to address the risk of overfitting from too many variables (see, e.g., Berg et al, 2014). It selects only 25 variables on average (over the 15 rolling regressions).…”

Global Liquidity, the Private Sector and Debt Sustainability in Sub‐Saharan Africa

“…In practice, the DSF has a limited ability to predict actual debt problems (IMF, : 7) . One possible change to address such failings is the more generalized introduction of a probabilistic approach (Berg et al., ) partially included for borderline cases after the DSF's 2012 review (IMF, ). The inclusion of more country‐specific variables in the underlying methodology that generates the risk rating — as opposed to relying on the broad averages of low‐income countries — seems sensible, although the complexity generated has raised concerns about usability (Martin, ), and the IMF () notes such variables were only sparsely utilized since the last review.…”