Measuring Complex State Policies: Pitfalls and Considerations, with an Application to Race and Welfare Policy

Abstract

Welfare policy is multidimensional because of the political compromises, competing goals, and federalist structure underpinning it. This complexity has hindered measurement and, therefore, the comparability of research on race and welfare policy. This paper describes a measurement strategy that is transparent, replicable, and attuned to matching the assumptions of statistical models to the policy process. We demonstrate that this strategy leads to more nuanced conclusions regarding the relationship between minority caseloads and the flexibility of state welfare policies. The strategy and recommendations are adaptable to research agendas that scholars bring to the comparative study of welfare in the U.S. states, countries, or other units—and to other complex policies enacted in federal systems.

Studying Immigrant Policy One Law at a Time

“…Are the results sensitive to analyst choices or are they relatively robust across a range of decisions? A substantial literature has worked to make sense of state-level policy in this area, and scholars differ in how they measure state immigrant policy activity (Boushey & Luedtke, 2011;Filindra, 2013;Hero & Preuhs, 2007;Marquez & Schraufnagel, 2013;Monogan, 2013a;Nicholson-Crotty & Nicholson-Crotty, 2011;Pham & Van, 2014;Reich & Barth, 2012;Rivera, 2014;Schildkraut, 2001;Ybarra, Sanchez, & Sanchez, 2016). Some studies aggregate over time to analyze cross-sectional variance only, others split welcoming and hostile laws to be considered separately, some use panel data to consider the overtime patterns, and finally some subdivide laws by policy subarea.…”

Studying Public Policy through Immigration Policy: Advances in Theory and Measurement

“…Best practices for weighting immigration policy modeling suggest that various laws and rules do not have the same substantive effect on targeted populations and thus should not be judged as equal (Bjerre et al, 2015). This point is echoed herein in the analysis by Plutzer et al (2019). At a minimum, researchers should take into account the material (and even psychological) impact that a law may have on the lives of noncitizens as well as the reach a law may have, that is, the likely number of people who may be affected by its provisions (Bjerre et al, 2015).…”

“…Aggregation decisions can influence the relative ranking of units within the index, whether those are countries, or U.S. states (Bjerre et al, ). Moreover, as Monogan (), Reich () and Plutzer et al () all demonstrate, aggregation choices can influence the correlations with various predictors and thus either overestimate or hide significant relationships (also see D'Urso, ). Cerna and Chou () add another real‐world perspective on how measurement counts, that is, affects policy decisions.…”

“…These methodological selections are defensible because they do not make broad assumptions about the commonality of driving forces behind “immigration” policies writ large (but, see Plutzer et al, ), but rather take the conservative approach to demonstrate different types of political, economic, or demographic effects on thin slices of policy outcomes. Here scholars confront a second trade‐off, as Monogan (, p. XXX) writes: “Often researchers will be interested not necessarily in whether a specific law is adopted but rather in what a state's overall immigration policy is.…”

“…Second, we collect three articles that take different perspectives on the question of how to measure immigration policy—in other words, to consider policy itself as the dependent variable. These include how to aggregate and rank policy scores (Bjerre, Römer, & Zobel, ), how to reconcile complexity alongside generality (Monogan, ), as well as pitfalls and other trade‐offs in immigration adjacent fields of race and welfare (Plutzer, Berkman, Honaker, Ojeda, & Whitesell, ). Third, we consider how theory development can benefit by incorporating the role of interest groups into our models (Collingwood et al, ), on the one hand, and considering the effects of immigration policy on attitudes and behaviors of majority and minority groups (Ybarra, Juárez Pérez, & Sanchez, ).…”