Does Prevalence Matter to Physicians in Estimating Post-test Probability of Disease? A Randomized Trial

Abstract

BACKGROUND: The probability of a disease following a diagnostic test depends on the sensitivity and specificity of the test, but also on the prevalence of the disease in the population of interest (or pre-test probability). How physicians use this information is not well known. OBJECTIVE:To assess whether physicians correctly estimate post-test probability according to various levels of prevalence and explore this skill across respondent groups. DESIGN: Randomized trial. PARTICIPANTS: Population-based sample of 1,361 physicians of all clinical specialties. INTERVENTION: We described a scenario of a highly accurate screening test (sensitivity 99% and specificity 99%) in which we randomly manipulated the prevalence of the disease (1%, 2%, 10%, 25%, 95%, or no information). MAIN MEASURES: We asked physicians to estimate the probability of disease following a positive test (categorized as <60%, 60-79%, 80-94%, 95-99.9%, and >99.9%). Each answer was correct for a different version of the scenario, and no answer was possible in the "no information" scenario. We estimated the proportion of physicians proficient in assessing post-test probability as the proportion of correct answers beyond the distribution of answers attributable to guessing. KEY RESULTS: Most respondents in each of the six groups (67%-82%) selected a post-test probability of 95-99.9%, regardless of the prevalence of disease and even when no information on prevalence was provided. This answer was correct only for a prevalence of 25%. We estimated that 9.1% (95% CI 6.0-14.0) of respondents knew how to assess correctly the post-test probability. This proportion did not vary with clinical experience or practice setting. CONCLUSIONS: Most physicians do not take into account the prevalence of disease when interpreting a positive test result. This may cause unnecessary testing and diagnostic errors.KEY WORDS: Bayes' theorem; predictive value of tests; prevalence; sensitivity and specificity; diagnosis; risk assessment; probability; evidence-based medicine.

Doctors and Patients’ Susceptibility to Framing Bias: A Randomized Trial

“…The doctor survey primarily explored doctors' opinions about policy issue and was approved by the Research Ethics Committee of the University Hospitals of Geneva 19,20 ; the patient survey, primarily a patient satisfaction survey, was exempted from full review 21,22 .…”

“…Previous studies have shown that doctors' understanding of various terms used in medical literature, such as relative risk, absolute risk, or the number needed to treat, differ considerably from an objective, criterion based assessment 24 . Similarly, most doctors misunderstand numerical data regarding test accuracy, regardless of whether they are presented as sensitivity and specificity or likelihood ratios 25 , and fail to use relevant numerical information, such as disease prevalence, when they interpret the results of diagnostic tests 20,26 . Were doctors less prone to framing bias than patients, it would be possible to simply warn doctors of the patients' limited capacity to interpret numerical risk data, and suggest various communication aids to improve patient understanding.…”

Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study

“…What evidence there is suggests that test performance is the main factor that informs their choice. It has been reported, however, that a substantial proportion of clinicians have an inaccurate understanding of test performance parameters and apply them inaccurately [126][127][128][129][130][131] and so it may be the case that choices are being based on false assumptions. Other factors mentioned in more than one study were the pretest probability of having the condition, as defined by patient characteristics, patient acceptance of the test and the costs involved in carrying out the test, which are factors that are not readily transferable to the search process.…”

The indication area of a diagnostic test. Part I—discounting gain and loss in diagnostic certainty

“…This information also depends on the prior [19,22]. Correct interpretation of these test results and appreciating the prior is notoriously hard to learn [26,27].…”