The Society for Implementation Research Collaboration Instrument Review Project: A methodology to promote rigorous evaluation

Abstract

BackgroundIdentification of psychometrically strong instruments for the field of implementation science is a high priority underscored in a recent National Institutes of Health working meeting (October 2013). Existing instrument reviews are limited in scope, methods, and findings. The Society for Implementation Research Collaboration Instrument Review Project’s objectives address these limitations by identifying and applying a unique methodology to conduct a systematic and comprehensive review of quantitative instruments assessing constructs delineated in two of the field’s most widely used frameworks, adopt a systematic search process (using standard search strings), and engage an international team of experts to assess the full range of psychometric criteria (reliability, construct and criterion validity). Although this work focuses on implementation of psychosocial interventions in mental health and health-care settings, the methodology and results will likely be useful across a broad spectrum of settings. This effort has culminated in a centralized online open-access repository of instruments depicting graphical head-to-head comparisons of their psychometric properties. This article describes the methodology and preliminary outcomes.MethodsThe seven stages of the review, synthesis, and evaluation methodology include (1) setting the scope for the review, (2) identifying frameworks to organize and complete the review, (3) generating a search protocol for the literature review of constructs, (4) literature review of specific instruments, (5) development of an evidence-based assessment rating criteria, (6) data extraction and rating instrument quality by a task force of implementation experts to inform knowledge synthesis, and (7) the creation of a website repository.ResultsTo date, this multi-faceted and collaborative search and synthesis methodology has identified over 420 instruments related to 34 constructs (total 48 including subconstructs) that are relevant to implementation science. Despite numerous constructs having greater than 20 available instruments, which implies saturation, preliminary results suggest that few instruments stem from gold standard development procedures. We anticipate identifying few high-quality, psychometrically sound instruments once our evidence-based assessment rating criteria have been applied.ConclusionsThe results of this methodology may enhance the rigor of implementation science evaluations by systematically facilitating access to psychometrically validated instruments and identifying where further instrument development is needed.Electronic supplementary materialThe online version of this article (doi:10.1186/s13012-014-0193-x) contains supplementary material, which is available to authorized users.

Intentional research design in implementation science: implications for the use of nomothetic and idiographic assessment

“…Implementation science also frequently focuses on the rigorous measurement of implementation outcomes (e.g., feasibility, fidelity, sustainment, cost; [4,29,30]) across multiple levels. Indeed, many models of implementation science and practice rely on targeted measurement of key variables, such as the organizational context, intervention acceptability, the quality of practitioner training and post-training supports, or the delivery of key intervention content [18,31,32].…”

“…Additionally, due to its multilevel nature, the assessment of implementation outcomes carries particular challenges [13]. For example, implementation measurement instruments oftentimes lack established psychometric properties or are impractical for use in community contexts [18,29,33].…”

“…Although both nomothetic and idiographic assessment approaches are used in the contemporary literature, it is rare that these approaches are explicitly defined, differentiated, or justified in the context of the research design, question, or assumptions and implications of findings. Most of the extant literature appears to reflect the nomothetic tradition, with a preponderance of research focused on the development and use of instruments intended to yield nomothetic data [29,34]. Virtually no guidance exists to support the intentional use of these two assessment approaches in implementation research, which provides opportunities for the strategic integration of nomothetic and idiographic data.…”

“…Instruments developed from a nomothetic tradition may be subject to proprietary rights and copyright restrictions. As a result, highly specialized or overly complicated assessment instruments may lack access or relevance within certain community settings [29]. In addition, less research has focused on the score validity of idiographic methods as compared to nomothetic, and very little research has investigated the combined use of both approaches.…”

“…Is the measure relevant to stakeholders? Will the data be actionable ORIGINAL RESEARCH [29]. These study considerations have clear implications for using nomothetic and/or idiographic approaches.…”