SeroTracker-RoB: a decision rule-based algorithm for reproducible risk of bias assessment of seroprevalence studies

Abstract

BackgroundConducting risk of bias assessments for seroprevalence studies is a crucial component of infection surveillance but can be a time-consuming and subjective process. We aimed to develop and evaluate decision rules for transparent and reproducible risk of bias assessments of seroprevalence studies.MethodsWe developed the SeroTracker-ROB decision rules, which generate risk of bias assessments for seroprevalence studies from an adapted version of the Joanna Briggs Institute Critical Appraisal Checklist for Prevalence Studies. The decision rules were developed using published guidance on risk of bias assessment for prevalence studies, and the consensus opinions of researchers that have critically appraised thousands of prevalence studies. The decision rules were evaluated against SeroTracker’s living systematic review database of SARS-CoV-2 seroprevalence studies. We determined decision rule coverage by calculating the proportion of database studies for which SeroTracker-ROB yielded a risk of bias assessment, and reliability by calculating intraclass correlations between SeroTracker-RoB assessments and the consensus manual judgements of two independent reviewers.ResultsThe SeroTracker-ROB decision rules for risk of bias assessment classified 100% (n = 2,070) of prevalence studies in SeroTracker’s database and showed good reliability compared to manual review (intraclass correlation 0.77, 95% CI 0.74 to 0.80). We developed a tool that implements these decision rules for use by other researchers.ConclusionsThe SeroTracker-ROB decision rules enabled rapid, transparent, and reproducible risk of bias assessment of seroprevalence studies, and may serve to support infection surveillance.

SARS-CoV-2 infection in Africa: A systematic review and meta-analysis of standardised seroprevalence studies, from January 2020 to December 2021

“…We evaluated study risk of bias using a validated, seroprevalence-specific tool based on the Joanna Briggs Institute (JBI) Checklist for seroprevalence studies,[17] and found that 37% of studies were at high risk of bias. The most common reasons for this included an unrepresentative sample frame, non-probability sampling, and not adjusting estimates for population characteristics.…”

“…evaluated overall risk of bias by summing each item’s score; the automated decision rule in our tool puts more weight on items more likely to bias results, such as whether a representative sample frame was chosen and a well-performing antibody test was used. [17]…”

Global SARS-CoV-2 seroprevalence: a systematic review and meta-analysis of standardized population-based studies from Jan 2020-May 2022

“…(20) This decision rule was developed based on guidance on estimating disease prevalence (21,22) and was validated against overall risk of bias assessments derived manually by two independent reviewers for previously collected seroprevalence studies in the SeroTracker database, showing good agreement with manual review (intraclass correlation 0.77, 95% CI 0.74-0.80; n = 2070 studies). (20)…”

“…(20) This decision rule was developed based on guidance on estimating disease prevalence (21,22) and was validated against overall risk of bias assessments derived manually by two independent reviewers for previously collected seroprevalence studies in the SeroTracker database, showing good agreement with manual review (intraclass correlation 0.77, 95% CI 0.74-0.80; n = 2070 studies). (20) We classified seroprevalence studies by geographical scope (local, sub-national, or national), sample frame, sampling method, and type of serological assay (Supplement S2.1, Table 5). Where multiple summary estimates were available per study, we prioritized estimates based on estimate adjustment, antibody isotypes measured, test type used, and antibody targets measured (full details: Supplement S3.1).…”

“…JBI checklist ratings were initially completed for each study by two independent reviewers, with conflicts resolved by consensus. To assess the overall risk of bias for studies included in this review, we used an automated decision rule that assigned a rating of low, moderate, or high risk of bias based on the specific combination of JBI checklist ratings for that study (20). This decision rule was developed based on guidance on estimating disease prevalence (21,22) and was validated against overall risk of bias assessments derived manually by two independent reviewers for previously collected seroprevalence studies in the SeroTracker database, showing good agreement with manual review (intraclass correlation 0.77, 95% CI 0.74-0.80; n = 2070 studies) (20).…”