PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies

Abstract

Background: Clinical prediction models combine several predictors (risk or prognostic factors) to estimate the risk whether a particular condition is present (diagnostic model) or whether a certain event will occur in the future (prognostic model). Large numbers of diagnostic and prognostic prediction model studies are published each year and a tool facilitating their quality assessment is needed, e.g. to support systematic reviews and evidence syntheses.Objective: To introduce and describe the development of PROBAST, a tool for assessing the risk of bias and applicability of prediction model studies.Methods: Web-based Delphi procedure (involving 40 experts in the field of prediction model research) and refinement of the tool through piloting. The scope of PROBAST was determined with consideration of existing risk of bias tools and reporting guidelines, such as CHARMS, QUADAS, QUIPS, and TRIPOD.Results: After seven Delphi rounds, a final tool was developed which utilises a domain-based structure supported by signalling questions. PROBAST assesses the risk of bias of prediction model studies and any concerns for their applicability. Studies that PROBAST can be used for include those developing, validating, and extending a prediction model. We define risk of bias to occur when shortcomings in the study design, conduct or analysis lead to systematically distorted estimates of model predictive performance or to an inadequate model to address the research question. The predictive performance is typically evaluated using calibration and discrimination, and sometimes (notably in diagnostic model studies) classification measures. Applicability refers to the extent to which the prediction model study matches the systematic review question in terms of the target population, predictors, or outcomes of interest. PROBAST comprises 20 signalling questions grouped into four domains: participant selection, predictors, outcome, and analysis.Conclusions: PROBAST can be used to assess the risk of bias and any concerns for applicability of studies developing, validating or extending (adjusting) prediction, both diagnostic and prognostic, models.

Risk scores for type 2 diabetes mellitus in Latin America: a systematic review of population‐based studies

“… PROBAST, Prediction model Risk of Bias ASsessment Tool ; +, low risk of bias/low concern regarding applicability; −, high risk of bias/high concern regarding applicability; ?, unclear risk of bias/unclear concern regarding applicability. …”

“…This is a sound methodological review following international guidelines for the systematic reviews of prognosis models . In addition, we used several search engines including ones based in Latin America, hence most, if not all, available evidence should have been retrieved.…”

“…Risk of bias was assessed following the PROBAST recommendations, a risk of bias assessment tool developed exclusively for prognosis models . This process was conducted by two reviewers independently (DJA‐G and JRM) and verified by a third (AB‐O).…”

On the aggregation of published prognostic scores for causal inference in observational studies

“…It is also possible to combine literature models, which are presented as score charts or risk scores (eg, TENOR risk score in our illustrative example), or which have been implemented in a web app. To critically appraise previously published prognostic models that may be relevant for aggregation, we recommend the use of PROBAST and CHARMS tools . Both were developed to help researchers assess the risk of bias and applicability of published prognostic models, and thus infer whether their inclusion would be beneficial.…”

A critical appraisal of the clinical applicability and risk of bias of the predictive models for mortality and recurrence in patients with oropharyngeal cancer: Systematic review

“…In order to be able to use the models that have been developed to stratify risk in patients with oropharyngeal cancer to select the optimal therapeutic strategy, it is necessary to determine whether the models present good standards that translate into low risk of bias and high clinical applicability . Therefore, the objective of this study was to carry out a systematic review of models for predicting mortality and recurrence in patients diagnosed with oropharyngeal cancer.…”