A systematic review of school health policy measurement tools: Implementation determinants and outcomes

Abstract

Background Governments in some countries or states/provinces mandate school-based policies intended to improve the health and well-being of primary and secondary students, and in some cases the health of school staff. Examples include mandating a minimum time spent per week in programmed physical activity, mandating provision of healthy foods and limiting fat content of school meals, and banning tobacco products or use on school campuses. Although school health researchers have studied whether schools, districts, or states/provinces are meeting requirements, it is unclear to what extent implementation processes and determinants are assessed. The purposes of the present systematic review of quantitative measures of school policy implementation are to identify quantitative measures of implementation determinants and Proctor model implementation outcomes and assess pragmatic and psychometric properties of such measures. Methods Peer-reviewed journal articles published 1995–2020 were included if they: 1) had multiple-item quantitative measures of school policy implementation; and 2) addressed overall wellness, tobacco, physical activity, nutrition, obesity prevention, and mental health/bullying/social-emotional learning. The final sample comprised 86 measurement tools from 67 peer-review articles. We extracted study characteristics, such as psychometric and pragmatic measure properties, from included articles based on three frameworks: Implementation Outcomes Framework, 2) Consolidated Framework for Implementation Research, and 3) Policy Implementation Determinants Framework. Results Most implementation tools were developed to measure overall wellness policies which combined multiple policy topics (n = 35, 40%) and were in survey form (n = 75, 87%). Fidelity was the most frequently prevalent implementation outcome (n = 70, 81%), followed by adoption (n = 32, 81%). The implementation determinants most assessed were readiness for implementation, including resources (n = 43, 50%), leadership (n = 42, 49%), and policy communication (n = 41, 48%). Overall, measures were low-cost and had easy readability. However, lengthy tools and lack of reported validity/reliability data indicate low transferability. Conclusions Implementation science can contribute to more complete and rigorous assessment of school health policy implementation processes, which can improve implementation strategies and ultimately the intended health benefits. Several high-quality measures of implementation determinants and implementation outcomes can be applied to school health policy implementation assessment. D&I researchers can also benefit from measurement experiences of school health researchers.

Quantitative measures used in empirical evaluations of mental health policy implementation: A systematic review

“…Overall, the observed pragmatic qualities aligned with previous findings (Allen et al, 2020; McLoughlin et al, 2021; Powell et al, 2021; Weiner et al, 2020). Most included measures were freely available (i.e., accessible through supplemental materials with the article or by contacting authors directly), though one was proprietary.…”

“…In the current review, we retrieved studies excluded during full-text screening by Allen et al, due to policy- or setting-specific measures ( n = 597). We then updated the original search strategy, with guidance from a research librarian, to include mental health-specific terms; a similar approach was used in previous reviews (McLoughlin et al, 2021; Walsh-Bailey et al, 2020). We ran the updated search in January 2021, using search strings related to mental health, public policy, implementation, and measurement (see Appendix B; full search strategy available in Appendices C and D).…”

“…Lastly, while systematic reviews have examined quantitative measures of implementation across public health topics (Chaudoir et al, 2013; Chor et al, 2015; Clinton-McHarg et al, 2016; Emmons et al, 2011; Kaplan et al, 2010; Weiner et al, 2008), less scholarship has been done related to measures of policy implementation. Recent work has examined generalizable measures of health policy implementation (Allen et al, 2020), as well as health policy measures targeting school settings (McLoughlin et al, 2021), healthy food environments (Phulkerd et al, 2016), and chronic disease (Walsh-Bailey et al, 2020). To our knowledge, no such study has been conducted for mental health policies.…”

The psychometric and pragmatic evidence rating scale (PAPERS) for measure development and evaluation

“…We present here the Psychometric and Pragmatic Evidence Rating Scale (PAPERS; Table 1). All or portions of PAPERS were applied across the seven systematic reviews in this special collection entitled, “Systematic Reviews of Methods to Measure Implementation Constructs.” Its uses extend beyond the collection (Allen et al, 2020; Gabriella et al, 2021; Khadjesari et al, 2020), however, to application in future systematic reviews and inform the development and selection of implementation measurement.…”