Variation in Research Designs Used to Test the Effectiveness of Dissemination and Implementation Strategies: A Review

Abstract

BackgroundThe need for optimal study designs in dissemination and implementation (D&I) research is increasingly recognized. Despite the wide range of study designs available for D&I research, we lack understanding of the types of designs and methodologies that are routinely used in the field. This review assesses the designs and methodologies in recently proposed D&I studies and provides resources to guide design decisions.MethodsWe reviewed 404 study protocols published in the journal Implementation Science from 2/2006 to 9/2017. Eligible studies tested the efficacy or effectiveness of D&I strategies (i.e., not effectiveness of the underlying clinical or public health intervention); had a comparison by group and/or time; and used ≥1 quantitative measure. Several design elements were extracted: design category (e.g., randomized); design type [e.g., cluster randomized controlled trial (RCT)]; data type (e.g., quantitative); D&I theoretical framework; levels of treatment assignment, intervention, and measurement; and country in which the research was conducted. Each protocol was double-coded, and discrepancies were resolved through discussion.ResultsOf the 404 protocols reviewed, 212 (52%) studies tested one or more implementation strategy across 208 manuscripts, therefore meeting inclusion criteria. Of the included studies, 77% utilized randomized designs, primarily cluster RCTs. The use of alternative designs (e.g., stepped wedge) increased over time. Fewer studies were quasi-experimental (17%) or observational (6%). Many study design categories (e.g., controlled pre–post, matched pair cluster design) were represented by only one or two studies. Most articles proposed quantitative and qualitative methods (61%), with the remaining 39% proposing only quantitative. Half of protocols (52%) reported using a theoretical framework to guide the study. The four most frequently reported frameworks were Consolidated Framework for Implementing Research and RE-AIM (n = 16 each), followed by Promoting Action on Research Implementation in Health Services and Theoretical Domains Framework (n = 12 each).ConclusionWhile several novel designs for D&I research have been proposed (e.g., stepped wedge, adaptive designs), the majority of the studies in our sample employed RCT designs. Alternative study designs are increasing in use but may be underutilized for a variety of reasons, including preference of funders or lack of awareness of these designs. Promisingly, the prevalent use of quantitative and qualitative methods together reflects methodological innovation in newer D&I research.

Designs and methods for implementation research: Advancing the mission of the CTSA program

“…Within-site designs are discussed in the section on quasi-experimental designs as they generally lack the replicability standard given their focus on one site or unit. It is important to acknowledge that other authors, such as Miller et al [35] and Mazzucca et al [36], have categorized certain designs somewhat differently than we have here.…”

Parenting Programs for Underserved Populations in Low- and Middle-Income Countries: Issues of Scientific Integrity and Social Justice

“…Our team also urges for treatment developers to think about the transportability and sustainability of their interventions for different settings. Designing and testing interventions using methods such as User Centered Design (Baumann et al, 2018;Lyon et al, 2014;Lyon & Kroner, 2016) could be a beneficial way to balance scientific integrity with social justice in LMIC.…”

“…Community engagement is multilevel (Brown et al, 2014;Mazzucca et al, 2018) and involves stakeholders in many roles. Particularly in global health, the concept of community and stakeholders need to be broaden, as we often face a lack of trained professionals able to deliver interventions (Belfer, 2008).…”

“…We encourage, therefore, researchers to think about different designs that may be able to accommodate issues of internal and internal validity (Landsverk et al, 2017), as well as practice, pragmatic and ethical considerations of RCTs when implementing EBPIs in LMICS (Brown et al, 2017;Curran, Bauer, Mittman, Pyne, & Stetler, 2012;Landsverk et al, 2017;Mazzucca et al, 2017). An overview of the different designs proposed by implementation scientists to support accelerating the reach of EBPIs is beyond the scope of this manuscript but can be found in other reviews (Landsverk et al, 2017;Mazzucca et al, 2018). The bi-directional communication among HIC and LMIC partners is not only beneficial for good practice of research grounded in social justice, but also to "bring back" lessons learned from LMIC.…”

Interventions to reduce unnecessary caesarean sections in healthy women and babies

“…Implementation and improvement science offers a range of theories to operationalise effective interventions. [102][103][104] Successful initiatives for complex health systems have used participatory methods and action-led processes to build audit and change cycles into the intervention process. 105,106 These implementation mechanisms allow for more flexible designs to integrate local barriers and mediators while still incorporating population-level evidence.…”