Variation of a test’s sensitivity and specificity with disease prevalence

Abstract

Research CMAJBackground: Anecdotal evidence suggests that the sensitivity and specificity of a diagnostic test may vary with disease prevalence. Our objective was to investigate the associations between disease prevalence and test sensitivity and specificity using studies of diagnostic accuracy. Methods:We used data from 23 meta-analyses, each of which included 10-39 studies (416 total). The median prevalence per review ranged from 1% to 77%. We evaluated the effects of prevalence on sensitivity and specificity using a bivariate random-effects model for each metaanalysis, with prevalence as a covariate. We estimated the overall effect of prevalence by pooling the effects using the inverse variance method.Results: Within a given review, a change in prevalence from the lowest to highest value resulted in a corresponding change in sensitivity or specificity from 0 to 40 percentage points. This effect was statistically significant (p < 0.05) for either sensitivity or specificity in 8 metaanalyses (35%). Overall, specificity tended to be lower with higher disease prevalence; there was no such systematic effect for sensitivity. Interpretation:The sensitivity and specificity of a test often vary with disease prevalence; this effect is likely to be the result of mechanisms, such as patient spectrum, that affect prevalence, sensitivity and specificity. Because it may be difficult to identify such mechanisms, clinicians should use prevalence as a guide when selecting studies that most closely match their situation. Abstract

Author’s response

“…The lack of significant associations may, in part, be due to the fact that the studies and the meta-analyses were not designed to investigate this issue and therefore by underpowered to detect it. Leeflang et al 5 discuss the potential reasons for these effects and highlight, as we did, that this cannot be interpreted as change in prevalence 'causes' change in sensitivity and specificity. Rather, in light of the empirical relationship observed, the effects may be due to other unmeasured factors or mechanisms that affect both prevalence and accuracy.…”

Diagnostic Accuracy of Fecal Calprotectin for Pediatric Inflammatory Bowel Disease in Primary Care: A Prospective Cohort Study

“…A meta-analysis showed that sensitivity remains stable over a range of prevalences and is not substantially influenced by spectrum bias. 37 A normal FCal level thus could be used to prevent a referral of children with functional symptoms to specialist care.…”

Clinical history for diagnosis of dementia in men: Caerphilly Prospective Study

“…The overall utility (AUC) of 'any memory difficulty' reported by either subject or informant was close to chance, suggesting that the sampling strategy oversampled men with cognitive impairment and hence would overestimate the PPV but would have underestimated the specificity. 32 Other investigators have found that subjective memory problems are inconsistently associated with dementia. 33 The estimates for diagnostic utility are likely to be overoptimistic as they result from stepwise selection procedures; some items may have been identified as being diagnostically useful by chance.…”