The psychometric and pragmatic evidence rating scale (PAPERS) for measure development and evaluation

Lewis, Cara C.; Mettert, Kayne D.; Halko, Heather M.; Nolen, Elspeth; Powell, Byron J.; Weiner, Bryan J.

doi:10.1177/26334895211037391

Cited by 37 publications

(73 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…managing or acquiring resources). In 4 of the 6 studies (66.7%) [ 59 , 61 , 68 , 79 ], the used champion subscale had acceptable internal consistency ( α ≥ 0.70 [ 97 ]); one study (16.7%) reported that the used champion subscale had low internal consistency ( α = 0.43) [ 50 ]. In 2 of the 6 studies (33.3%), the authors conducted an exploratory factor analysis and reported that the champion items loaded highly to a single factor [ 68 , 77 , 78 ].…”

Section: Resultsmentioning

confidence: 99%

The effectiveness of champions in implementing innovations in health care: a systematic review

Santos

Graham

Lalonde

et al. 2022

Implement Sci Commun

View full text Add to dashboard Cite

Background Champions have been documented in the literature as an important strategy for implementation, yet their effectiveness has not been well synthesized in the health care literature. The aim of this systematic review was to determine whether champions, tested in isolation from other implementation strategies, are effective at improving innovation use or outcomes in health care. Methods The JBI systematic review method guided this study. A peer-reviewed search strategy was applied to eight electronic databases to identify relevant articles. We included all published articles and unpublished theses and dissertations that used a quantitative study design to evaluate the effectiveness of champions in implementing innovations within health care settings. Two researchers independently completed study selection, data extraction, and quality appraisal. We used content analysis and vote counting to synthesize our data. Results After screening 7566 records titles and abstracts and 2090 full text articles, we included 35 studies in our review. Most of the studies (71.4%) operationalized the champion strategy by the presence or absence of a champion. In a subset of seven studies, five studies found associations between exposure to champions and increased use of best practices, programs, or technological innovations at an organizational level. In other subsets, the evidence pertaining to use of champions and innovation use by patients or providers, or at improving outcomes was either mixed or scarce. Conclusions We identified a small body of literature reporting an association between use of champions and increased instrumental use of innovations by organizations. However, more research is needed to determine causal relationship between champions and innovation use and outcomes. Even though there are no reported adverse effects in using champions, opportunity costs may be associated with their use. Until more evidence becomes available about the effectiveness of champions at increasing innovation use and outcomes, the decision to deploy champions should consider the needs and resources of the organization and include an evaluation plan. To further our understanding of champions’ effectiveness, future studies should (1) use experimental study designs in conjunction with process evaluations, (2) describe champions and their activities and (3) rigorously evaluate the effectiveness of champions’ activities. Registration Open Science Framework (https://osf.io/ba3d2). Registered on November 15, 2020.

show abstract

Section: Resultsmentioning

confidence: 99%

The effectiveness of champions in implementing innovations in health care: a systematic review

Santos

Graham

Lalonde

et al. 2022

Implement Sci Commun

View full text Add to dashboard Cite

show abstract

“…The pragmatic and psychometric evidence of each eligible measure was assessed and scored using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS) [ 17 , 31 ]. PAPERS includes 14 items that assesses nine psychometric properties and five pragmatic features (see Table 1 ).…”

Section: Methodsmentioning

confidence: 99%

“…Measures should also illustrate reliability and evidence of other domains of validity (e.g., concurrent validity), to ensure accuracy and reduce error. Finally, measures should exhibit important pragmatic qualities, including easy access, use, scoring, and interpretation [ 17 ]. Pragmatic qualities are less frequently evaluated but are essential in ensuring the uptake of reliable and valid measures.…”

Section: Introductionmentioning

confidence: 99%

Evaluation of measures of sustainability and sustainability determinants for use in community, public health, and clinical settings: a systematic review

et al. 2022

View full text Add to dashboard Cite

Background Sustainability is concerned with the long-term delivery and subsequent benefits of evidence-based interventions. To further this field, we require a strong understanding and thus measurement of sustainability and what impacts sustainability (i.e., sustainability determinants). This systematic review aimed to evaluate the quality and empirical application of measures of sustainability and sustainability determinants for use in clinical, public health, and community settings. Methods Seven electronic databases, reference lists of relevant reviews, online repositories of implementation measures, and the grey literature were searched. Publications were included if they reported on the development, psychometric evaluation, or empirical use of a multi-item, quantitative measure of sustainability, or sustainability determinants. Eligibility was not restricted by language or date. Eligibility screening and data extraction were conducted independently by two members of the research team. Content coverage of each measure was assessed by mapping measure items to relevant constructs of sustainability and sustainability determinants. The pragmatic and psychometric properties of included measures was assessed using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). The empirical use of each measure was descriptively analyzed. Results A total of 32,782 articles were screened from the database search, of which 37 were eligible. An additional 186 publications were identified from the grey literature search. The 223 included articles represented 28 individual measures, of which two assessed sustainability as an outcome, 25 covered sustainability determinants and one explicitly assessed both. The psychometric and pragmatic quality was variable, with PAPERS scores ranging from 14 to 35, out of a possible 56 points. The Provider Report of Sustainment Scale had the highest PAPERS score and measured sustainability as an outcome. The School-wide Universal Behaviour Sustainability Index-School Teams had the highest PAPERS score (score=29) of the measure of sustainability determinants. Conclusions This review can be used to guide selection of the most psychometrically robust, pragmatic, and relevant measure of sustainability and sustainability determinants. It also highlights that future research is needed to improve the psychometric and pragmatic quality of current measures in this field. Trial registration This review was prospectively registered with Research Registry (reviewregistry1097), March 2021.

show abstract

“…This study's strengths include the use of established measure development best practices ( 27 , 28 , 39 ), engagement of key partners, including target end users, in each development stage, and transparent reporting of early findings so that they may be used by other researchers and practitioners ( 56 ). This study contributes to filling a gap in the measure development methods literature in that we provide detail on processes for identifying relevant constructs and engaging target audiences in developing and refining measures often not detailed in the literature.…”

Section: Discussionmentioning

confidence: 99%

Investigating implementation of school health policies through a health equity lens: A measures development study protocol

McLoughlin

Walsh‐Bailey

Singleton

et al. 2022

Front. Public Health

View full text Add to dashboard Cite

BackgroundSchool-based policies that ensure provision of nutrition, physical activity, and other health-promoting resources and opportunities are essential in mitigating health disparities among underserved populations. Measuring the implementation of such policies is imperative to bridge the gap between policy and practice. Unfortunately, limited practical, psychometrically strong measures of school policy implementation exist. Few available explicitly focus on the issues of equity and social justice as a key component of implementation, which may result in underassessment of the equity implications of policy implementation. The purpose of this study is to develop equity-focused measures in collaboration with practitioners, researchers, and other key implementation partners that will facilitate evaluation of policy implementation determinants (i.e., barriers and facilitators), processes, and outcomes.MethodsWe will actively seek engagement from practitioners, researchers, and advocacy partners (i.e., stakeholders) who have expertise in school health policy throughout each phase of this project. We propose a multi-phase, 1-year project comprising the following steps: (1) selection of relevant constructs from guiding frameworks related to health equity and implementation science; (2) initial measure development, including expert feedback on draft items; (3) pilot cognitive testing with representatives from key target populations (i.e., school administrators, teachers, food service staff, and students and parents/guardians); and (4) measure refinement based on testing and assessment of pragmatic properties. These steps will allow us to establish initial face and content validity of a set of instruments that can undergo psychometric testing in future studies to assess their reliability and validity.DiscussionCompletion of this project will result in several school policy implementation measurement tools which can be readily used by practitioners and researchers to evaluate policy implementation through a health equity lens. This will provide opportunities for better assessment and accountability of policies that aim to advance health equity among school-aged children and their families.Trial registrationOpen Science Framework Registration doi: 10.17605/OSF.IO/736ZU.

show abstract

The psychometric and pragmatic evidence rating scale (PAPERS) for measure development and evaluation

Cited by 37 publications

References 15 publications

The effectiveness of champions in implementing innovations in health care: a systematic review

The effectiveness of champions in implementing innovations in health care: a systematic review

Evaluation of measures of sustainability and sustainability determinants for use in community, public health, and clinical settings: a systematic review

Investigating implementation of school health policies through a health equity lens: A measures development study protocol

Contact Info

Product

Resources

About