The Sensitivity of Country Ranks to Index Construction and Aggregation Choice: The Case of Immigration Policy

Abstract

How sensitive are country ranks to the aggregation function used in index construction? This paper tests whether different aggregation functions come to different results in regard to the ranking of countries. Indices within the field of immigration and integration policy are analyzed, yet, the results pertain to index building across the social sciences. The paper discusses three aggregation methods: the arithmetic mean, the geometric mean, and a noncompensatory/non‐linear aggregation function based on the Condorcet method. In the empirical part, these three aggregation functions are applied to the family indicators for the year 2010 of the Immigration Policies in Comparison (IMPIC) dataset, a new dataset which measures immigration policies’ restrictiveness, as well as to the eight policy strands of the Migrant Integration Policy Index for the year 2014. Results show that the methods react differently to extreme values and thus result in different rank orders in the middle range. In the politicized field of immigration and integration policies, country ranks play a crucial role and this is shown to have profound real‐world implications. The paper thus urges researchers to be reflective of the assumptions of different aggregation functions, as these lead to different results.

Measuring Complex State Policies: Pitfalls and Considerations, with an Application to Race and Welfare Policy

“…The simplest approach is to simply “add ‘em up” by counting each element equally (equivalent to assigning all included rules a weight of “1” and any excluded rules a weight of “0”). Unit weighting has a long tradition in state politics research (e.g., Squire’s index of legislative professionalism), partly because “add ‘em up” scales are usually highly correlated with measures that assign magnitudes to the weights (e.g., see Bowen & Greene, ; and nearly the same as calculating the geometric mean, see Bjerre et al, ). Unit weighting does require some subjective decision making by the investigator, however.…”

“…This large body of literature, however, raises the question of how state welfare policy should be measured. The challenge of measuring a complex policy is the subject of many papers in this special issue (Bjerre, Römer, & Zobel, ; Goodman, ; Monogan, ; Reich, ). These papers grapple with more general questions of comparative policy measurement and most especially questions concerning whether specific aspects of policy should be combined into summary scores that measure legislative output.…”

Studying Public Policy through Immigration Policy: Advances in Theory and Measurement

“…Simple, additive aggregation is often the main type used, though other procedures exist and can produce different kinds of rankings and results (Bjerre et al, ; Plutzer et al, ). Aggregation decisions can influence the relative ranking of units within the index, whether those are countries, or U.S. states (Bjerre et al, ). Moreover, as Monogan (), Reich () and Plutzer et al () all demonstrate, aggregation choices can influence the correlations with various predictors and thus either overestimate or hide significant relationships (also see D'Urso, ).…”

“…Aggregation decisions are a significant, if often overlooked, component of immigration research design. Simple, additive aggregation is often the main type used, though other procedures exist and can produce different kinds of rankings and results (Bjerre et al, ; Plutzer et al, ). Aggregation decisions can influence the relative ranking of units within the index, whether those are countries, or U.S. states (Bjerre et al, ).…”

“…Second, we collect three articles that take different perspectives on the question of how to measure immigration policy—in other words, to consider policy itself as the dependent variable. These include how to aggregate and rank policy scores (Bjerre, Römer, & Zobel, ), how to reconcile complexity alongside generality (Monogan, ), as well as pitfalls and other trade‐offs in immigration adjacent fields of race and welfare (Plutzer, Berkman, Honaker, Ojeda, & Whitesell, ). Third, we consider how theory development can benefit by incorporating the role of interest groups into our models (Collingwood et al, ), on the one hand, and considering the effects of immigration policy on attitudes and behaviors of majority and minority groups (Ybarra, Juárez Pérez, & Sanchez, ).…”