The risk of bias in observational studies of exposures (ROBINS-E) tool: concerns arising from application to observational studies of exposures

Abstract

BackgroundSystematic reviews, which assess the risk of bias in included studies, are increasingly used to develop environmental hazard assessments and public health guidelines. These research areas typically rely on evidence from human observational studies of exposures, yet there are currently no universally accepted standards for assessing risk of bias in such studies. The risk of bias in non-randomised studies of exposures (ROBINS-E) tool has been developed by building upon tools for risk of bias assessment of randomised trials, diagnostic test accuracy studies and observational studies of interventions. This paper reports our experience with the application of the ROBINS-E tool.MethodsWe applied ROBINS-E to 74 exposure studies (60 cohort studies, 14 case-control studies) in 3 areas: environmental risk, dietary exposure and drug harm. All investigators provided written feedback, and we documented verbal discussion of the tool. We inductively and iteratively classified the feedback into 7 themes based on commonalities and differences until all the feedback was accounted for in the themes. We present a description of each theme.ResultsWe identified practical concerns with the premise that ROBINS-E is a structured comparison of the observational study being rated to the ‘ideal’ randomised controlled trial. ROBINS-E assesses 7 domains of bias, but relevant questions related to some critical sources of bias, such as exposure and funding source, are not assessed. ROBINS-E fails to discriminate between studies with a single risk of bias or multiple risks of bias. ROBINS-E is severely limited at determining whether confounders will bias study outcomes. The construct of co-exposures was difficult to distinguish from confounders. Applying ROBINS-E was time-consuming and confusing.ConclusionsOur experience suggests that the ROBINS-E tool does not meet the need for an international standard for evaluating human observational studies for questions of harm relevant to public and environmental health. We propose that a simpler tool, based on empirical evidence of bias, would provide accurate measures of risk of bias and is more likely to meet the needs of the environmental and public health community.

Making nutrition guidelines fit for purpose

“…Tools for assessing human observational studies of exposure effects need further development 1213. A recent review of 62 tools for assessing observational studies of exposures could not recommend a specific tool but provided guidance for selecting one 14.…”

Impact of Alcohol Policies on Suicidal Behavior: A Systematic Literature Review

“…Even fewer adjusted for other influences on suicide rates, such as age, gender/sex, and socio-economic deprivation distributions. However, the assessment of risk of bias in these studies is complicated by the lack of clear guidance on evaluating bias in studies of exposures [ 43 ].…”

Does physical activity strengthen lungs and protect against asthma in childhood? A systematic review

“…The NOS has drawn some criticism for its potential to produce arbitrary results . Similarly, the more detailed ROBINS‐E tool has been criticized for being time‐consuming, for confusing, and for failing to assess some sources of bias . Additionally, the small number of included studies meant that further analyses to investigate effect potential modification by factors such as age bracket and asthma status could not be performed.…”