Inclusion of Nonrandomized Studies of Interventions in Systematic Reviews of Intervention Effectiveness: An Update

Abstract

Introduction: Nonrandomized studies of interventions (NRSIs) are observational or experimental studies of the effectiveness and/or harms of interventions, in which participants are not randomized to intervention groups. There is increasingly widespread recognition that advancements in the design and analysis of NRSIs allow NRSI evidence to have a much more prominent role in decision making, and not just as ancillary evidence to randomized controlled trials (RCTs). Objective: To guide decisions about inclusion of NRSIs for addressing the effects of interventions in systematic reviews (SRs), this chapter updates the 2010 guidance on inclusion of NRSIs in Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) SRs. The chapter focuses on considerations for decisions to include or exclude NRSIs in SRs. Methods: In November 2020, AHRQ convened a 20-member workgroup that comprised 13 members representing 8 of 9 AHRQ-appointed EPCs, 3 AHRQ representatives, 1 independent consultant with expertise in SRs, and 3 representatives of the AHRQ-appointed Scientific Resource Center. The workgroup received input from the full EPC Program regarding the process and specific issues through discussions at a virtual meeting and two online surveys regarding challenges with NRSI inclusion in SRs. One survey focused on current practices by EPCs regarding NRSI inclusion in ongoing and recently completed SRs. The other survey focused on the appropriateness, completeness, and usefulness of existing EPC Program methods guidance. The workgroup considered the virtual meeting and survey input when identifying aspects of the guidance that needed updating. The workgroup used an informal method for generating consensus about guidance. Disagreements were resolved through discussion. Results: We outline considerations for the inclusion of NRSIs in SRs of intervention effectiveness. We describe the strengths and limitations of RCTs, study design features and types of NRSIs, and key considerations for making decisions about inclusion of NRSIs (during the stages of topic scoping and refinement, SR team formation, protocol development, SR conduct, and SR reporting). We discuss how NRSIs may be applicable for the decisional dilemma being addressed in the SR, threats to the internal validity of NRSIs, as well as various data sources and advanced analytic methods that may be used in NRSIs. Finally, we outline an approach to incorporating NRSIs within an SR and key considerations for reporting. Conclusion: The main change from the previous guidance is the overall approach to decisions about inclusion of NRSIs in EPC SRs. Instead of recommending NRSI inclusion only if RCTs are insufficient to address the Key Question, this updated guidance handles NRSI evidence as a valuable source of information and lays out important considerations for decisions about the inclusion of NRSIs in SRs of intervention effectiveness. Different topics may require different decisions regarding NRSI inclusion. This guidance is intended to improve the utility of the final product to end-users. Inclusion of NRSIs will increase the scope, time, and resources needed to complete SRs, and NRSIs pose potential threats to validity, such as selection bias, confounding, and misclassification of interventions. Careful consideration must be given to both concerns.

Development of a Health Equity Framework for the US Preventive Services Task Force

“…Identifying and ranking other outcomes of importance to mitigate inequities may necessitate soliciting public or patient input, particularly from groups experiencing a disproportionate burden of disease. Inclusion of well-conducted nonrandomized studies of interventions (NRSIs) for the effectiveness of clinical services could be considered even when robust randomized clinical trial (RCT) evidence exists, as RCTs may focus on narrowly defined populations and there may be concern about extrapolating trial findings to populations experiencing health disparities . In these instances, the rationale for and limitations of inclusion of NRSIs should be articulated.…”

Effectiveness of mHealth interventions to promote physical activity and reduce sedentary behaviours on work-related productivity and performance: a systematic review protocol

“…Studies conducted as randomised controlled trials (RCTs) or non-randomised studies of interventions (NRSI) with a predefined control/comparator group (eg, non-randomized controlled trials, cohort studies and pre-post studies). 25 This review will include studies with an NRSI design based on a scientific overview of occupational health interventions, which encouraged researchers to include alternatives to the RCT design when conducting systematic reviews on this topic. 26 This approach will provide a broad body of evidence to guide the implementation of interventions in real-world settings in which it would be difficult or inappropriate to conduct an RCT.…”

“…Studies conducted as randomised controlled trials (RCTs) or non-randomised studies of interventions (NRSI) with a predefined control/comparator group (eg, non-randomized controlled trials, cohort studies and pre-post studies). 25 This review will include studies with an NRSI design based on a scientific overview of occupational health interventions, which encouraged…”

Conducting separate reviews of benefits and harms could improve systematic reviews and meta-analyses

“…Even systematic reviews and meta-analyses of multiple randomized trials are typically underpowered to detect uncommon and rare events [58]. Non-randomized studies can provide better estimates of differences in uncommon harms, harms that occur after prolonged exposure, harms with long latency (e.g., occurring after acute treatment or after treatment discontinuation), and harms that occur in target populations who use interventions outside of trials [59][60][61][62][63][64]. "Real-world evidence" is used increasingly to evaluate the benefis and harms of interventions, including data from electronic health records, claims, and surveillance systems [64][65][66][67][68][69].…”

“…Non-randomized studies can provide better estimates of differences in uncommon harms, harms that occur after prolonged exposure, harms with long latency (e.g., occurring after acute treatment or after treatment discontinuation), and harms that occur in target populations who use interventions outside of trials [59][60][61][62][63][64]. "Real-world evidence" is used increasingly to evaluate the benefis and harms of interventions, including data from electronic health records, claims, and surveillance systems [64][65][66][67][68][69]. For new drugs, pre-clinical studies also provide valuable evidence about harms that are difficult to observe in people, such as effects on developing embryos and drug interactions.…”