Assessing risk of bias in human environmental epidemiology studies using three tools: different conclusions from different tools

Abstract

Background Systematic reviews are increasingly prevalent in environmental health due to their ability to synthesize evidence while reducing bias. Different systematic review methods have been developed by the US National Toxicology Program’s Office of Health Assessment and Translation (OHAT), the US Environmental Protection Agency’s (EPA) Integrated Risk Information System (IRIS), and by the US EPA under the Toxic Substances Control Act (TSCA), including the approach to assess risk of bias (ROB), one of the most vital steps which is used to evaluate internal validity of the studies. Our objective was to compare the performance of three tools (OHAT, IRIS, TSCA) in assessing ROB. Methods We selected a systematic review on polybrominated diphenyl ethers and intelligence quotient and/or attention deficit hyperactivity disorder because it had been endorsed by the National Academy of Sciences. Two reviewers followed verbatim instructions from the tools and independently applied each tool to assess ROB in 15 studies previously identified. We documented the time to apply each tool and the impact the ROB ratings for each tool had on the final rating of the quality of the overall body of evidence. Results The time to complete the ROB assessments varied widely (mean = 20, 32, and 40 min per study for the OHAT, IRIS, and TSCA tools, respectively). All studies were rated overall “low” or “uninformative” using IRIS, due to “deficient” or “critically deficient” ratings in one or two domains. Similarly, all studies were rated “unacceptable” using the TSCA tool because of one “unacceptable” rating in a metric related to statistical power. Approximately half of the studies had “low” or “probably low ROB” ratings across all domains with the OHAT and Navigation Guide tools. Conclusions Tools that use overall ROB or study quality ratings, such as IRIS and TSCA, may reduce the available evidence to assess the harms of environmental exposures by erroneously excluding studies, which leads to inaccurate conclusions about the quality of the body of evidence. We recommend using ROB tools that circumvents these issues, such as OHAT and Navigation Guide. Systematic review registration This review has not been registered as it is not a systematic review.

Integrated Risk Information System (IRIS) response to “Assessing risk of bias in human environmental epidemiology studies using three tools: different conclusions from different tools”

“…First, Eick et al [ 1 ] reported that the IRIS study evaluation approach resulted in low confidence or uninformative ratings in all studies; however, we did not reach the same conclusions with our analysis of the same studies. As described in the IRIS Handbook, the study evaluation process begins with development and pilot testing exposure- and outcome-specific criteria that identify the information and appropriate methods needed to apply the evaluation ratings in each domain (for examples, see [ 5 ]).…”

“…“Assessing risk of bias in human environmental epidemiology studies using three tools: different conclusions from different tools” by Eick et al [ 1 ] applied the study evaluation approach developed by the U.S. Environmental Protection Agency’s Integrated Risk Information System (IRIS), as well as other approaches, to a set of studies examining polybrominated diphenyl ethers (PBDEs) and neurodevelopment. They concluded that use of the IRIS approach resulted in excessive exclusion of studies, which would lead to hazard conclusions based on an incomplete body of evidence.…”

“…These criteria are based on the state of knowledge about the toxicokinetics of the chemical being assessed, exposure assessment methods, and the epidemiological standard of practice for specific outcomes. They improve transparency and consistency of our reviews, and were not part of the Eick et al [ 1 ] evaluation. Importantly, in addition to risk of bias, the IRIS study evaluation approach considers the sensitivity of a study to detect effects if they exist, which improves our ability to interpret study findings.…”

“…Importantly, in addition to risk of bias, the IRIS study evaluation approach considers the sensitivity of a study to detect effects if they exist, which improves our ability to interpret study findings. To illustrate how the approach is used by IRIS scientists, we re-applied the IRIS tool to the same 10 studies of PBDE exposure and IQ as in Eick et al [ 1 ].…”

Acute and chronic exposure to air pollution in relation with incidence, prevalence, severity and mortality of COVID-19: a rapid systematic review

“…As recently reported by a panel of experts [ 24 , 25 ]; currents risk of bias (RoB) tools tend to dilute the quality of study assessing adverse effects of environmental exposures. This occurs by incorrectly excluding studies contributing to misleading the quality of the body of evidence.…”