Let’s Measure Agreement, Consensus, and Polarization in Ordinal Data

Abstract

Different measures exist to capture agreement, consensus, concentration, dispersion, and polarization in ordinal data. We compare consensus scores across specific situations for a better understanding of how different measures work in practice: constructed cases, simulated data where we know the underlying distribution, and empirical data. Although researchers have solved the ‘problem’ of measuring agreement, consensus, and polarization several times, we highlight similarities and equivalence across some existing approaches, while others differ substantially. The choice of method can lead to substantively different conclusions, and we recommend that researchers use a combination of measures and use graphics to examine the distribution qualitatively.