Knowledge user survey and Delphi process to inform development of a new risk of bias tool to assess systematic reviews with network meta-analysis (RoB NMA tool)

Abstract

BackgroundNetwork meta-analysis (NMA) is increasingly used in guideline development and other aspects of evidence-based decision-making. We aimed to develop a risk of bias (RoB) tool to assess NMAs (RoB NMA tool). An international steering committee recommended that the RoB NMA tool to be used in combination with the Risk of Bias in Systematic reviews (ROBIS) tool (i.e. because it was designed to assess biases only) or other similar quality appraisal tools (eg, A MeaSurement Tool to Assess systematic Reviews 2 [AMSTAR 2]) to assess quality of systematic reviews. The RoB NMA tool will assess NMA biases and limitations regarding how the analysis was planned, data were analysed and results were presented, including the way in which the evidence was assembled and interpreted.ObjectivesConduct (a) a Delphi process to determine expert opinion on an item’s inclusion and (b) a knowledge user survey to widen its impact.DesignCross-sectional survey and Delphi process.MethodsDelphi panellists were asked to rate whether items should be included. All agreed-upon item were included in a second round of the survey (defined as 70% agreement). We surveyed knowledge users’ views and preferences about the importance, utility and willingness to use the RoB NMA tool to evaluate evidence in practice and in policymaking. We included 12 closed and 10 open-ended questions, and we followed a knowledge translation plan to disseminate the survey through social media and professional networks.Results22 items were entered into a Delphi survey of which 28 respondents completed round 1, and 22 completed round 2. Seven items did not reach consensus in round 2. A total of 298 knowledge users participated in the survey (14% respondent rate). 75% indicated that their organisation produced NMAs, and 78% showed high interest in the tool, especially if they had received adequate training (84%). Most knowledge users and Delphi panellists preferred a tool to assess both bias in individual NMA results and authors’ conclusions. Response bias in our sample is a major limitation as knowledge users working in high-income countries were more represented. One of the limitations of the Delphi process is that it depends on the purposive selection of experts and their availability, thus limiting the variability in perspectives and scientific disciplines.ConclusionsThis Delphi process and knowledge user survey informs the development of the RoB NMA tool.

Decision makers find it difficult to compare and select similar systematic reviews based on quality, methods and results: a cross-sectional survey

“…Reviews that intend to compare multiple treatments with an NMA but then nd that the expectations or assumptions are violated (e.g. the network is 'disconnected', studies are too heterogeneous to combine, underlying assumptions of the method are not met), and hence an NMA is not possible or optimal, are also considered in our de nition [24].…”

Methodological review of NMA bias concepts provides groundwork for the development of a list of concepts for potential inclusion in a new risk of bias tool for network meta-analysis (RoB NMA Tool)

“…The concepts in Appendix 3 were categorised into the following domains: 3 concepts in network characteristics, 4 concepts in effect modi ers, 13 concepts in statistical synthesis, and 2 concepts in interpretation of the ndings and conclusions. These concepts should not be used to assess bias in NMAs as they are preliminary thoughts which will be altered and re ned into items based on expert feedback [91].…”

“…This study represents the rst stage in the development of a new risk of bias tool for NMAs. This systematic review of items identi ed 22 concepts which were entered into a Delphi survey to solicit expert opinion [91]. The steering committee used the expert feedback to choose and re ne the concepts.…”

“…The steering committee used the expert feedback to choose and re ne the concepts. We also considered feedback from a stakeholder survey on the structure, conceptual decisions, and concepts in the proposed tool [91]. The concepts were then be worded into items and an elaboration and explanation document was written.…”

Exploring advanced methods for network meta-analysis

“…Lunny et al share the findings from their stakeholder and Delphi surveys, which aimed to gather insights into the preferences and interests of respondents regarding the focus and items to be incorporated into their forthcoming risk of bias (RoB) tool for assessing SRs with NMA (RoB NMA tool) 12…”