When to Consult Precision-Recall Curves

Cook, Jonathan; Ramadas, Vikram

doi:10.2139/ssrn.3350582

Cited by 12 publications

(14 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Receiver operating characteristic (ROC) curves were plotted using Xpert test results as the reference standard and areas under the curve (ROC AUCs) were calculated. In addition, we calculated the area under the precision-recall curve (PR AUC), due to the test library’s low overall Xpert positivity rate 48 . We then identified two cut-off thresholds to transform the continuous abnormality score of each CAD software into dichotomous normal/abnormal interpretations that matched the sensitivity achieved by the Expert and Intermediate Readers.…”

Section: Methodsmentioning

confidence: 99%

Independent evaluation of 12 artificial intelligence solutions for the detection of tuberculosis

Codlin¹,

Dao²,

Vo³

et al. 2021

Sci Rep

View full text Add to dashboard Cite

There have been few independent evaluations of computer-aided detection (CAD) software for tuberculosis (TB) screening, despite the rapidly expanding array of available CAD solutions. We developed a test library of chest X-ray (CXR) images which was blindly re-read by two TB clinicians with different levels of experience and then processed by 12 CAD software solutions. Using Xpert MTB/RIF results as the reference standard, we compared the performance characteristics of each CAD software against both an Expert and Intermediate Reader, using cut-off thresholds which were selected to match the sensitivity of each human reader. Six CAD systems performed on par with the Expert Reader (Qure.ai, DeepTek, Delft Imaging, JF Healthcare, OXIPIT, and Lunit) and one additional software (Infervision) performed on par with the Intermediate Reader only. Qure.ai, Delft Imaging and Lunit were the only software to perform significantly better than the Intermediate Reader. The majority of these CAD software showed significantly lower performance among participants with a past history of TB. The radiography equipment used to capture the CXR image was also shown to affect performance for some CAD software. TB program implementers now have a wide selection of quality CAD software solutions to utilize in their CXR screening initiatives.

show abstract

Section: Methodsmentioning

confidence: 99%

Independent evaluation of 12 artificial intelligence solutions for the detection of tuberculosis

Codlin¹,

Dao²,

Vo³

et al. 2021

Sci Rep

View full text Add to dashboard Cite

show abstract

“…Nonetheless, Fawcett et al 35 advocated the use of ROC because it is insensitive to changes in the prevalence of the outcome. Cook and Ramadas 36 explained that if the primary goal, as relevant to pharmacovigilance, is to maximize sensitivity, by identifying all of the positive cases, ROC curves may still be preferable.…”

Section: Discussionmentioning

confidence: 99%

Comparison of laboratory threshold criteria in drug‐induced liver injury detection algorithms for use in pharmacovigilance

Tan

Ling

Ang

et al. 2020

Pharmacoepidemiology and Drug

View full text Add to dashboard Cite

Purpose: For the purpose of pharmacovigilance, we sought to determine the best performing laboratory threshold criteria to detect drug-induced liver injury (DILI) in the electronic medical records (EMR). Methods: We compared three commonly used liver chemistry criteria from the DILI expert working group (DEWG), DILI network (DILIN), and Council for International Organizations of Medical Sciences (CIOMS), based on hospital EMR for years 2010 and 2011 (42 176 admissions), using independent medical record review. The performance characteristics were compared in terms of sensitivity, specificity, positive predictive value (PPV), negative predictive value, accuracy, F-measure, and area under the receiver operating characteristic curve (AUROC).

show abstract

“…Similar to the ROC AUC score, the area under the PR curve (PR AUC) summarizes the model's performance, where values closer to one indicate better classifiers. As Cook and Ramadas [39] states, the ROC curve is preferred when we care more about identifying a high percentage of the positives, whereas the PR AUC ensures that the positive predicted instances are mainly positive. Here we employ both metrics as they provide somewhat complementary information about model's predictive potential.…”

Section: Predictive Potentialmentioning

confidence: 99%

A Hybrid Ensemble Feature Selection Design for Candidate Biomarkers Discovery from Transcriptome Profiles

Colombelli¹,

Kowalski²,

Recamonde‐Mendoza³

2021

Preprint

View full text Add to dashboard Cite

The discovery of disease biomarkers from gene expression data has been greatly advanced by feature selection (FS) methods, especially using ensemble FS (EFS) strategies with perturbation at the data level (i.e., homogeneous, Hom-EFS) or method level (i.e., heterogeneous, Het-EFS). Here we proposed a Hybrid EFS (Hyb-EFS) design that explores both types of perturbation to improve the stability and the predictive power of candidate biomarkers. With this, Hyb-EFS aims to disrupt associations of good performance with a single dataset, single algorithm, or a specific combination of both, which is particularly interesting for better reproducibility of genomic biomarkers. We investigated the adequacy of our approach for microarray data related to four types of cancer, carrying out an extensive comparison with other ensemble and single FS approaches. Five FS methods were used in our experiments: Wx, Symmetrical Uncertainty (SU), Gain Ratio (GR), Characteristic Direction (GeoDE), and ReliefF. We observed that the Hyb-EFS and Het-EFS approaches attenuated the large performance variation observed for most single FS and Hom-EFS across distinct datasets. Also, the Hyb-EFS improved upon the stability of the Het-EFS within our domain. Comparing the Hyb-EFS and Het-EFS composed of the top-performing selectors (Wx, GR, and SU), our hybrid approach surpassed the equivalent heterogeneous design and the best Hom-EFS (Hom-Wx). Interestingly, the rankings produced by our Hyb-EFS reached greater biological plausibility, with a notably high enrichment for cancer-related genes and pathways. Thus, our experiments suggest the potential of the proposed Hybrid EFS design in discovering candidate biomarkers from microarray data. Finally, we provide an open-source framework to support similar analyses in other domains, both as a user-friendly application and a plain Python package.

show abstract

When to Consult Precision-Recall Curves

Cited by 12 publications

References 17 publications

Independent evaluation of 12 artificial intelligence solutions for the detection of tuberculosis

Independent evaluation of 12 artificial intelligence solutions for the detection of tuberculosis

Comparison of laboratory threshold criteria in drug‐induced liver injury detection algorithms for use in pharmacovigilance

A Hybrid Ensemble Feature Selection Design for Candidate Biomarkers Discovery from Transcriptome Profiles

Contact Info

Product

Resources

About