Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey

Abstract

BackgroundDecisionmakers and guideline developers demand rapid syntheses of the evidence when time sensitive evidence-informed decisions are required. A potential trade-off of such rapid reviews is that their results can have less reliability than results of systematic reviews that can lead to an increased risk of making incorrect decisions or recommendations. We sought to determine how much incremental uncertainty about the correctness of an answer guideline developers and health policy decisionmakers are willing to accept in exchange for a rapid evidence-synthesis.MethodsEmploying a purposive sample, we conducted an international web-based, anonymous survey of decisionmakers and guideline developers. Based on a clinical treatment, a public health, and a clinical prevention scenario, participants indicated the maximum risk of getting an incorrect answer from a rapid review that they would be willing to accept. We carefully reviewed data and performed descriptive statistical analyses.ResultsIn total, 325 (58.5%) of 556 participants completed our survey and were eligible for analysis. The median acceptable incremental risk for getting an incorrect answer from a rapid review across all three scenarios was 10.0% (interquartile range [IQR] 5.0–15.0). Acceptable risks were similar for the clinical treatment (n = 313, median 10.0% [IQR 5.0–15.0]) and the public health scenarios (n = 320, median 10.0% [IQR 5.0–15.0]) and lower for the clinical prevention scenario (n = 312, median 6.5% [IQR 5.0–10.5]).ConclusionsFindings suggest that decisionmakers are willing to accept some trade-off in validity in exchange for a rapid review. Nevertheless, they expect the validity of rapid reviews to come close to that of systematic reviews.Electronic supplementary materialThe online version of this article (doi:10.1186/s12874-017-0406-5) contains supplementary material, which is available to authorized users.

Prioritising references for systematic reviews with RobotAnalyst: A user study

“…For example, the ability to prioritise and screen by clusters is an interesting scheme that may be useful for scoping reviews, or as a way to organise and assess a preliminary literature search strategy. In the case of a rapid review, it may not be as essential to finish screening once enough relevant evidence has been identified. Other use‐cases include performing review updates or deciding if a review needs to be updated .…”

Redefining rapid reviews: a flexible framework for restricted systematic reviews

“…A survey of decision makers and guideline developers about how much uncertainty they would accept when using restricted reviews suggests that there is a willingness to accept some possible trade-off in internal validity in exchange for timeliness 10. It should be noted that systematic reviews have their own limitations, on which we do not reflect here, and these limitations therefore also apply to restricted reviews 11…”

Current methods for development of rapid reviews about diagnostic tests: an international survey

“…Performing selected review activities simultaneously e.g., data extraction of known studies while a search of new studies is conducted risk for an inaccurate answer in exchange for a rapid product; thus, current RR developers would be reluctant to compromise the validity of results in exchange for implementation of methodological shortcuts and limits [37,38]. We acknowledge several potential limitations in this study.…”