Measuring Electoral Democracy with Observables

Abstract

Most crossnational indices of democracy rely centrally on coder judgments, which are susceptible to personal bias and error, and also require expensive and time-consuming coding by experts. The few measures based exclusively on observable indicators are either dichotomous or rely on a few rather crude proxies. This project lays out an approach to measurement based on observables that aims to preserve the nuanced quality of subjectively coded democracy indices. First, we gather data for a wide range of observable indicators, X´, that capture different aspects of the democratic process. Next, we use supervised random forest machine learning to predict Z using factual indicators, X´, creating an observable-to-subjective score mapping (OSM). The mapping that provides the best cross-validated fit to the outcome serves as an alternate index, Z´, for that conceptualization of democracy.Information loss from Z to Z´ is minimal for indices centered on an electoral conception of democracy and this loss may be advantageous for some purposes. It is free of idiosyncratic coder errors arising from misinformation, slack, or biases for or against a regime. It is also less susceptible to systematic bias that may arise from coders’ inferences about a country’s regime status, e.g., from the ideology of the current ruler. The data collection procedure and mode of analysis is fully transparent and replicable, and the procedure is cheap to produce, easy to update, and offers coverage for all polities with sovereign or semisovereign status, surpassing the sample of any existing index. We show that this expansive coverage makes a big difference to our understanding of some causal questions.

Measuring Backsliding with Observables: Observable-to-Subjective Score Mapping

“…Our approach enlists OSM, an approach to measurement adapted for situations in which both subjectively coded and directly observable ("objective") indicators of a concept are available (Weitzel et al 2023a). We begin with extant indices of democracy based largely on subjective coding, with the assumption that these measurement instruments have some prima facie validity-or at least did have before the era of alleged backsliding.…”

“…As long as a feature was observable for a broad set of cases and potentially relevant to democracy, it was included in our canvas. A total of 26 indicators drawn from Weitzel et al (2023a) was reenlisted for our study (see table SI 1.1 in the online appendix).…”

“…We first define our methodology, called observable-to-subjective score mapping (OSM) and based on Weitzel et al (2023a), which includes further details. The second section presents the results of our analyses applied to three prominent democracy indices.…”

Measuring Democratic Backsliding

“…Nevertheless, changes in the average index can provide a general sense of how indicators related to the quality of democracy have changed over time. For an approach that attempts to approximate democracy scores using only objective indicators, see Weitzel et al (2023).…”