Effects of dependent errors in the assessment of diagnostic test performance

Torrance-Rynard, Vicki; Walter, Stephen D.

doi:10.1002/(sici)1097-0258(19971015)16:19<2157::aid-sim653>3.0.co;2-x

Cited by 132 publications

(102 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…There are, however, some important concerns with latent class modelling. There are, for example, important concerns linked to the identifiability of the latent class model (Rindskopf and Rindskopf, 1986;Uebersax and Grove, 1990) and especially the dependence model used (Torrance-Rynard and Walter, 1997;Albert and Dodd, 2004;Albert et al, 2001 Pepe andJanes, 2007). Indeed it must be explicitly recognised that the latent class modelling approach is not a panacea for ground reference data problems.…”

mentioning

confidence: 77%

Assessing the accuracy of land cover change with imperfect ground reference data

Foody

2010

Remote Sensing of Environment

279

159

View full text Add to dashboard Cite

The manuscript of the above article revised after peer review and submitted to the journal for publication, follows. Please note that small changes may have been made after submission and the definitive version is that subsequently published as: AbstractThe ground data used as a reference in the validation of land cover change products are often not an ideal gold standard but degraded by error. The effects of ground reference data error on the accuracy of land cover change detection and the accuracy of estimates of the extent of change was evaluated. Twelve data sets were simulated to allow the exploration of the impacts of a spectrum of ground data imperfections on the estimation of the producer's and user's accuracy of change as well as of change extent. Simulated data were used since this ensured that the actual properties of the data were known and to exclude effects due to other sources of ground reference data error; although the impacts of simulated reference data on two real confusion matrices is also illustrated. The imperfections evaluated ranged from the inclusion of small amounts of known error into the ground reference data through to the extreme situation in which ground data were absent. The results show that even small amounts of error in the ground reference data can introduce large error into studies of land cover change by remote sensing and reinforce the desire to avoid the expression ground truth as this might imply that the data are a gold standard reference. The effect of reference data imperfections was dependent on the degree of association between the errors in the cross tabulated data sets. For 3 example, in the scenarios investigated, a 10% error in the reference data set introduced an under-estimation of the producer's accuracy of 18.5% if the errors were independent but an over-estimation of the producer's accuracy of 12.3% if the errors were correlated. The magnitude of the mis-estimation of the producer's accuracy was also a function of the amount of change and greatest at low levels of change. The amount of land cover change estimated also varied greatly as a function of ground reference data error. Some possible methods to reduce or even remove the impacts of ground reference data error were illustrated. These ranged from simple algebraic means to estimate the actual values of accuracy and change extent if the imperfections were known through to a latent class analysis that allowed the assessment of classification accuracy and estimation of change extent without the use of ground reference data if the underlying model is defined appropriately.

show abstract

mentioning

confidence: 77%

Assessing the accuracy of land cover change with imperfect ground reference data

Foody

2010

Remote Sensing of Environment

279

159

View full text Add to dashboard Cite

show abstract

“…We also studied the impact of adjusting this model for conditional dependence, i.e., a correlation between BinaxNOW-SP and the reference test within the groups of S. pneumoniae-positive and -negative individuals in each study. We considered models with different degrees of correlation (18) and compared them by using the deviance information criterion (19).…”

Section: Methodsmentioning

confidence: 99%

Systematic Review and Meta-Analysis of a Urine-Based Pneumococcal Antigen Test for Diagnosis of Community-Acquired Pneumonia Caused by Streptococcus pneumoniae

et al. 2013

View full text Add to dashboard Cite

Standard culture methods for diagnosis of Streptococcus pneumoniae pneumonia take at least 24 h. The BinaxNOW urine-based test for S. pneumoniae (BinaxNOW-SP) takes only 15 min to conduct, potentially enabling earlier diagnosis and targeted treatment. This study was conducted to assess whether the use of BinaxNOW-SP at the time of hospital admission would provide adequate sensitivity and specificity for diagnosis of community-acquired pneumonia (CAP) in adult patients. We searched PubMed, EMBASE/OVID, Cochrane Collaboration, Centre for Reviews and Dissemination, INAHTA, and CADTH for diagnostic or etiologic studies of hospitalized predominately adult patients with clinically defined CAP that reported the diagnostic performance of BinaxNOW-SP versus cultures. Two authors independently extracted study details and diagnostic two-by-two tables. We found that 27 studies met our inclusion criteria, and three different reference standards were used between them. A bivariate meta-analysis of 12 studies using a composite of culture tests as the reference standard estimated the sensitivity of BinaxNOW-SP as 68.5% (95% credibility interval [CrI], 62.6% to 74.2%) and specificity as 84.2% (95% CrI, 77.5% to 89.3%). A meta-analysis of all 27 studies, adjusting for the imperfect and variable nature of the reference standard, gave a higher sensitivity of 74.0% (CrI, 66.6% to 82·3%) and specificity of 97.2% (CrI, 92.7% to 99.8%). The analysis showed substantial heterogeneity across studies, which did not decrease with adjustment for covariates. We concluded that the higher pooled sensitivity (compared to culture) and high specificity of BinaxNOW-SP suggest it would be a useful addition to the diagnostic workup for community-acquired pneumonia. More research is needed regarding the impact of BinaxNOW-SP on clinical practice. Streptococcus pneumoniae pneumonia is believed to be the most common cause of community-acquired pneumonia (CAP) in adults, which in turn is the most common infection-related cause of death in developed countries (1). Diagnosis is usually established by observation of S. pneumoniae in a Gram-stained sputum sample or growth of S. pneumoniae in a culture of blood, sputum, pleural fluid, or other respiratory sample. Although highly specific, culturing is known to be insensitive, with diagnostic yields reported to be Ͻ30% for blood culture (2-7) and 57% for sputum culture (the latter in patients who had an etiologic diagnosis established) (8). Cultures also require 24 h or more to produce results. In the absence of a reliable rapid test for pneumonia caused by S. pneumoniae, initial treatment of pneumonia must be empirical, based upon knowledge of local pathogens, patient risk factors and comorbidities, and severity of presentation (7). Empirical therapy is generally effective (7), but there is increasing interest in improved targeting of antibiotics, due to the understanding that this may decrease the community prevalence of antibiotic-resistant bacteria and individual risks of antibiotic-associated Clostridium ...

show abstract

“…Such a dependence could occur, for instance, between tests which are based on the same underlying principle or between results from the same test at two different points in time. Several authors have demonstrated that it is important to account for this dependence while analyzing the results from diagnostic tests, in order to obtain unbiased estimates of the prevalence of disease and test parameters (Fryback, 1978;Vacek, 1985;Brenner, 1996; Torrance-Rynard and Walter, 1997). While there has been much recent work in this area, the problem of how to analyze diagnostic test data when the tests are correlated and when there is no perfect (gold standard) test remains to be solved, especially when the number of tests available is less than 5, as is usually the case in practice.…”

mentioning

confidence: 99%

Bayesian Approaches to Modeling the Conditional Dependence Between Multiple Diagnostic Tests

2001

View full text Add to dashboard Cite

show abstract

Effects of dependent errors in the assessment of diagnostic test performance

Cited by 132 publications

References 4 publications

Assessing the accuracy of land cover change with imperfect ground reference data

Assessing the accuracy of land cover change with imperfect ground reference data

Systematic Review and Meta-Analysis of a Urine-Based Pneumococcal Antigen Test for Diagnosis of Community-Acquired Pneumonia Caused by Streptococcus pneumoniae

Bayesian Approaches to Modeling the Conditional Dependence Between Multiple Diagnostic Tests

Contact Info

Product

Resources

About