Improving Understanding of Diagnostic Test Outcomes

Talboy, Alaina N.; Schneider, Sandra L.

doi:10.1177/0272989x18758293

Cited by 7 publications

(22 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Neither physicians nor patients are good at calculating PPV, although formats that provide test accuracy in terms of frequencies rather than probabilities improve this, at least for physicians. 11,13,18 People do even better assessing PPV when given information that conditions upon test results rather than health status, 31 although determining that information requires prevalence. In this study, analysis of GPs' reasoning suggests they assess the ''gist'' of the situation rather than work out exact calculations, consistent with how people make decisions in other settings.…”

Section: Discussionmentioning

confidence: 99%

Physician and Nonphysician Estimates of Positive Predictive Value in Diagnostic v. Mass Screening Mammography: An Examination of Bayesian Reasoning

Austin

2019

Med Decis Making

View full text Add to dashboard Cite

Background. The same test with the same result has different positive predictive values (PPVs) for people with different pretest probability of disease. Representative thinking theory suggests people are unlikely to realize this because they ignore or underweight prior beliefs when given new information (e.g., test results) or due to confusing test sensitivity (probability of positive test given disease) with PPV (probability of disease given positive test). This research examines whether physicians and MBAs intuitively know that PPV following positive mammography for an asymptomatic woman is less than PPV for a symptomatic woman and, if so, whether they correctly perceive the difference. Design. Sixty general practitioners (GPs) and 84 MBA students were given 2 vignettes of women with abnormal (positive) mammography tests: 1 with prior symptoms (diagnostic test), the other an asymptomatic woman participating in mass screening (screening test). Respondents estimated pretest and posttest probabilities. Sensitivity and specificity were neither provided nor elicited. Results. Eighty-eight percent of GPs and 46% of MBAs considered base rates and estimated PPV in diagnosis greater than PPV in screening. On average, GPs estimated a 27-point difference and MBAs an 18-point difference, compared to actual of 55 or more points. Ten percent of GPs and 46% of MBAs ignored base rates, incorrectly assessing the 2 PPVs as equal. Conclusions. Physicians and patients are better at intuitive Bayesian reasoning than is suggested by studies that make test accuracy values readily available to be confused with PPV. However, MBAs and physicians interpret a positive in screening as more similar to a positive in diagnosis than it is, with nearly half of MBAs and some physicians wrongly equating the two. This has implications for overdiagnosis and overtreatment.

show abstract

Section: Discussionmentioning

confidence: 99%

Physician and Nonphysician Estimates of Positive Predictive Value in Diagnostic v. Mass Screening Mammography: An Examination of Bayesian Reasoning

Austin

2019

Med Decis Making

View full text Add to dashboard Cite

show abstract

“…The feasibility of communicating and understanding these statistical concepts using best practices for risk communication 10–12 was demonstrated in a previous pilot study. 13 Levels for each attribute were selected to include those of commonly published prediction models.…”

Section: Methodsmentioning

confidence: 99%

Preferences for Predictive Model Characteristics among People Living with Chronic Lung Disease: A Discrete Choice Experiment

et al. 2020

View full text Add to dashboard Cite

Background. Patients may find clinical prediction models more useful if those models accounted for preferences for false-positive and false-negative predictive errors and for other model characteristics. Methods. We conducted a discrete choice experiment to compare preferences for characteristics of a hypothetical mortality prediction model among community-dwelling patients with chronic lung disease recruited from 3 clinics in Philadelphia. This design was chosen to allow us to quantify “exchange rates” between different characteristics of a prediction model. We provided previously validated educational modules to explain model attributes of sensitivity, specificity, confidence intervals (CI), and time horizons. Patients reported their interest in using prediction models themselves or having their physicians use them. Patients then chose between 2 hypothetical prediction models each containing varying levels of the 4 attributes across 12 tasks. Results. We completed interviews with 200 patients, among whom 95% correctly chose a strictly dominant model in an internal validity check. Patients’ interest in predictive information was high for use by themselves ( n = 169, 85%) and by their physicians ( n = 184, 92%). Interest in maximizing sensitivity and specificity were similar (0.88 percentage points of specificity equivalent to 1 point of sensitivity, 95% CI 0.72 to 1.05). Patients were willing to accept a reduction of 6.10 months (95% CI 3.66 to 8.54) in the predictive time horizon for a 1% increase in specificity. Discussion. Patients with chronic lung disease can articulate their preferences for the characteristics of hypothetical mortality prediction models and are highly interested in using such models as part of their care. Just as clinical care should become more patient centered, so should the characteristics of predictive models used to guide that care.

show abstract

“…These types of tests are commonly presented as Bayesian reasoning problems, which are used to evaluate the ability to update prior beliefs based on additional evidence to determine a posterior probability. Research over the last 40 years demonstrates that uninitiated or novice reasoners tend to have difficulty determining the PPV (e.g., Gigerenzer and Hoffrage, 1995 ; Gigerenzer et al, 2007 ; Reyna and Brainerd, 2008 ; Hoffrage et al, 2015 ; Johnson and Tubau, 2015 , 2017 ; Sirota et al, 2015 ; Talboy and Schneider, 2017 , 2018a , b ).…”

Section: Introductionmentioning

confidence: 99%

“…The goal of the present research is to extend our previous efforts to identify factors that prevent reasoners from being ready to recognize and apply Bayes theorem to update probabilities based on diagnostic test information (e.g., Talboy and Schneider, 2018a , b ). To do this, we deconstruct the problem into a simpler format that allows an assessment of whether and when reasoners can recognize and apply the needed information to infer the PPV.…”

Section: Introductionmentioning

confidence: 99%

“…Factors associated with the low accuracy rates observed in Bayesian reasoning tasks are broadly categorized as either representational or computational difficulties ( Johnson and Tubau, 2015 , 2017 ; Talboy and Schneider, 2018a , b ). Difficulties with how the problem is cognitively represented by reasoners are typically attributed to how the components of the problem relate to each other, which is more or less apparent depending on the formulation of the problem.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Reference Dependence in Bayesian Reasoning: Value Selection Bias, Congruence Effects, and Response Prompt Sensitivity

Talboy

Schneider

2022

Front. Psychol.

View full text Add to dashboard Cite

This work examines the influence of reference dependence, including value selection bias and congruence effects, on diagnostic reasoning. Across two studies, we explored how dependence on the initial problem structure influences the ability to solve simplified precursors to the more traditional Bayesian reasoning problems. Analyses evaluated accuracy and types of response errors as a function of congruence between the problem presentation and question of interest, amount of information, need for computation, and individual differences in numerical abilities. Across all problem variations, there was consistent and strong evidence of a value selection bias in that incorrect responses almost always conformed to values that were provided in the problem rather than other errors including those related to computation. The most consistent and unexpected error across all conditions in the first experiment was that people were often more likely to utilize the superordinate value (N) as part of their solution rather than the anticipated reference class values. This resulted in a weakened effect of congruence, with relatively low accuracy even in congruent conditions, and a dominant response error of the superordinate value. Experiment 2 confirmed that the introduction of a new sample drew attention away from the provided reference class, increasing reliance on the overall sample size. This superordinate preference error, along with the benefit of repeating the PPV reference class within the question, demonstrated the importance of reference dependence based on the salience of information within the response prompt. Throughout, higher numerical skills were generally associated with higher accuracy, whether calculations were required or not.

show abstract

Improving Understanding of Diagnostic Test Outcomes

Cited by 7 publications

References 41 publications

Physician and Nonphysician Estimates of Positive Predictive Value in Diagnostic v. Mass Screening Mammography: An Examination of Bayesian Reasoning

Physician and Nonphysician Estimates of Positive Predictive Value in Diagnostic v. Mass Screening Mammography: An Examination of Bayesian Reasoning

Preferences for Predictive Model Characteristics among People Living with Chronic Lung Disease: A Discrete Choice Experiment

Reference Dependence in Bayesian Reasoning: Value Selection Bias, Congruence Effects, and Response Prompt Sensitivity

Contact Info

Product

Resources

About