Different Types of Data and the Validity of Democracy Measures

Abstract

Different measures of democracy rely on different types of data. Some exclusively rely on observational data, others rely on judgement-based data in the form of in-house coded indicators or expert surveys. A third set of democracy measures combines information from indicators based on different types of data, some of them also data from representative surveys of the mass public. This article discusses the advantages and disadvantages of these different types of data for the measurement of electoral and liberal democracy. The discussion is based on the premise that the main priorities must be to establish a high degree of concept-measure consistency, i.e. indicators capture relevant aspects of the core concept of interest in a precise and unbiased manner, and to provide high coverage. The basic argument of the article is that no type of data is superior to others in all respects. The article draws on examples from extant datasets to illustrate the tradeoffs and it offers suggestions about how to reduce some of the potential drawbacks.

Is there a decline of democracy in the EU between 2004 and 2016? The relevance of data selection: a replication study of Smolka (2021) and comparison of democracy measures

“…In addition to the adequacy, the concept validity of the measurement must of course be included in the reasoning, which precedes the results of the measurements. This strand of literature took off with the seminal article by Munck and Verkuilen (2002), and has since then seen major improvements (inconclusive list Pickel et al 2015;Skaaning 2018;McMann et al 2021), although there are still some unresolved issues. More importantly, however, users will rely more heavily on these studies than they have in the past when selecting an appropriate measurement tool.…”

Why Choice Matters: Revisiting and Comparing Measures of Democracy

“…Second, the combination of different data sources or types promises to overcome the limitations inherent to each of them. This holds true at all stages of the research process as shown by the contributions of Elff and Ziaja (2018) and Skaaning (2018). Finally, the contributions by Fleuß et al (2018) as well as Landman (2018) highlight how the application of certain more maximalist definitions of democracy could change the comparison of different democratic regimes significantly.…”

Varieties of Measurement: A Comparative Assessment of Relatively New Democracy Ratings based on Original Data

“…This risk increases exponentially when enlisting public opinion surveys; while such surveys can help capture the relevant experiences of ordinary people, citizen responses about abstract beliefs are not a solid source for democracy measurement. All of the coding strategies can be biased in different ways due to limited access to relevant material, personal characteristics, and methodrelated factors influencing the filtering and processing of information (Bollen & Paxton 2000;Skaaning 2018).…”

“…Consider here that previous examinations have revealed significant variation among democracy datasets. These differences concern their focus, their reliability and validity, and -by implication -their correlates (see, e.g., Coppedge & Gerring et al 2011;Elff & Ziaja 2018;Skaaning 2018).…”