A systematic review: Tools for assessing methodological quality of human observational studies

Abstract

This review is, in press, to be published in the Australian National Health and Medical Research Council’s (NHMRC) Guidelines for Guidelines. Background: Observational studies are the primary human study type for evaluating potential harm from environmental exposures, and therefore critical for developing public health guidance. Although there are tools for assessing methodological quality in observational studies, there is no consensus on optimal tools or key elements to include when assessing quality. This study aims to help decision makers in selecting tools to assess risk of bias in observational studies of exposures. Objective: To identify, describe, categorize elements into domains, and evaluate published tools designed to assess methodological quality or risk of bias in observational human studies assessing the effects of exposures. Study appraisal and synthesis: Data was extracted on the characteristics, development and testing of each tool. The categories of items were classified into 9 domains informed by the literature. Results: We identified 62 tools with 17 categories of similar or overlapping items. Conclusions: Our review highlights the need for a common tool for assessing risk of bias in human observational studies of exposures. Absent that common tool, this review provides a clear cross-walk across available tools and identifies four main take home messages: (1) the tool should have clear definitions for each item and be transparent regarding the empirical or theoretical basis for each domain, (2) tools should include questions addressing 9 domains: Selection, Exposure, Outcome assessment, Confounding, Loss to follow-up, Analysis, Selective reporting, Conflicts of interest and Other, (3) the ratings for each domain should be reported, rather than an overall score, (4) the tool should be rigorously and independently tested for usability and reliability. Systematic Review Registration: PROSPERO: CRD42018094173

All-Cause and Cause-Specific Mortality Among People Using Extramedical Opioids

“…Because risk-of-bias tools for observational epidemiological studies are still evolving, 17 we developed a review-specific tool with reference to 2 recent publications 17,18 on assessing risk of bias in observational studies of exposures. The tool assessed each study on 2 risk-of-bias domains and 3 quality-ofreporting domains (eAppendix 3 in the Supplement).…”

Rapid Reviews in Zeiten von COVID-19 – Erfahrungen im Zuge des Kompetenznetzes Public Health zu COVID-19 und Vorschlag eines standardisierten Vorgehens

“…Als "klassische" Tools für randomisierte kontrollierte Studien ist das Cochrane RoB-Tool 2.0 [11] einsetzbar, für Kohorten-und Fallkontrollstudien zu Interventionen ROBINS-I [12], für diagnostische Studien QUADAS 2 [13]. Auch für den Einsatz in ätiologisch ausgerichteten Beobachtungsstudien ist eine hohe Zahl von Bewertungstools verfügbar: In einem systematischen Review unter Leitung von Wissenschaftler:innen der University of Sydney [14] [19]. Über die Beurteilung der Qualität der eingeschlossenen Primärstudien hinaus ist eine studienübergreifende Beurteilung der Vertrauenswürdigkeit der Evidenz wünschenswert, wie sie mit dem GRADE-Verfahren geleistet werden kann [20].…”

“…Als „klassische“ Tools für randomisierte kontrollierte Studien ist das Cochrane RoB-Tool 2.0 11 einsetzbar, für Kohorten- und Fallkontrollstudien zu Interventionen ROBINS-I 12 , für diagnostische Studien QUADAS 2 13 . Auch für den Einsatz in ätiologisch ausgerichteten Beobachtungsstudien ist eine hohe Zahl von Bewertungstools verfügbar: In einem systematischen Review unter Leitung von Wissenschaftler:innen der University of Sydney 14 konnten diesbezüglich 62 unterschiedliche Tools identifiziert werden. Die Entscheidung für ein spezifisches Bewertungstool kann in Abhängigkeit vom Einsatzbereich und Studienthema, aber auch von Vorerfahrungen der Reviewgruppe erfolgen.…”

The Nordic Nutrition Recommendations 2022 – structure and rationale of qualified systematic reviews

“…There is a lack of strong empirical evidence that one way of assessing the risk of bias is superior. Hence, many different approaches exist (42). The use of quality scales (e.g.…”

“…The use of quality scales (e.g. the often used Jadad scale for trials), in which a summary, numeric quality score is calculated, may be misleading, even causing contradictory conclusions about effects, and is discouraged (29,(42)(43)(44)(45). It has become more common to use and classify study quality simply as 'good', 'poor' or 'fair', or similar.…”