Area under the Precision-Recall Curve: Point Estimates and Confidence Intervals

Boyd, Kendrick; Eng, Kevin H.; Page, C. David

doi:10.1007/978-3-642-40994-3_29

Cited by 317 publications

(257 citation statements)

References 19 publications

Supporting

Mentioning

253

Contrasting

Unclassified

Order By: Relevance

“…While the loss function was deliberately designed to weigh recall higher than precision (at β = 0.7), consistent improvements in all test performance metrics including DSC and F 2 scores on the test set indicate improved generalization through this type of training. Compared to DSC which weighs recall and precision equally, and the ROC analysis, we consider the area under the PR curves (APR, shown in Figure 2) the most reliable performance metric for such highly skewed data [8,1]. To put the work in context, we reported average DSC, F 2 , and APR scores (equal to 56.4, 57.3, and 56.0, respectively), which indicate that our approach performed very well compared to the latest results in MS lesion segmentation [6,20].…”

Section: Resultsmentioning

confidence: 99%

“…To critically evaluate the performance of the detection for the highly unbalanced (skewed) dataset, we use the Precision-Recall (PR) curve (as opposed to the receiver-operator characteristic, or ROC, curve) as well as the area under the PR curve (the APR score) [1,7,8]. For such skewed datasets, the PR curves and APR scores (on test data) are preferred figures of algorithm performance.…”

Section: Evaluation Metricsmentioning

confidence: 99%

“…Tversky index is a generalization of the Dice similarity coefficient and the F β scores. We show how adjusting the hyperparameters of this index allow placing emphasis on false negatives in training a network that generalizes and performs well in highly imbalanced data as it leads to high sensitivity, Dice, F 2 score, and the area under the precision-recall (PR) curve [1] in the test set. To this end, we adopt a 3D FCN, based on the U-net architecture, with a Tversky loss layer, and test it in the challenging multiple sclerosis lesion detection problem on multi-channel MRI [6,20].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Tversky Loss Function for Image Segmentation Using 3D Fully Convolutional Deep Networks

Salehi

Erdoğmuş

Gholipour

2017

Lecture Notes in Computer Science

735

380

View full text Add to dashboard Cite

Abstract. Fully convolutional deep neural networks carry out excellent potential for fast and accurate image segmentation. One of the main challenges in training these networks is data imbalance, which is particularly problematic in medical imaging applications such as lesion segmentation where the number of lesion voxels is often much lower than the number of non-lesion voxels. Training with unbalanced data can lead to predictions that are severely biased towards high precision but low recall (sensitivity), which is undesired especially in medical applications where false negatives are much less tolerable than false positives. Several methods have been proposed to deal with this problem including balanced sampling, two step training, sample re-weighting, and similarity loss functions. In this paper, we propose a generalized loss function based on the Tversky index to address the issue of data imbalance and achieve much better trade-off between precision and recall in training 3D fully convolutional deep neural networks. Experimental results in multiple sclerosis lesion segmentation on magnetic resonance images show improved F2 score, Dice coefficient, and the area under the precision-recall curve in test data. Based on these results we suggest Tversky loss function as a generalized framework to effectively train deep neural networks.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Evaluation Metricsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Tversky Loss Function for Image Segmentation Using 3D Fully Convolutional Deep Networks

Salehi

Erdoğmuş

Gholipour

2017

Lecture Notes in Computer Science

735

380

View full text Add to dashboard Cite

show abstract

“…We calculated precisionrecall curves by successively filtering out results based on abundances to increase precision and recalculating recall at each step, defining true and false positives in terms of the binary detection of species. The AUPR was calculated using the lower trapezoid method [69]. For subspecies, classification at varying levels complicated the analysis (e.g.…”

Section: Precision-recallmentioning

confidence: 99%

Comprehensive Benchmarking and Ensemble Approaches for Metagenomic Classifiers

McIntyre

Ounit

Afshinnekoo

et al. 2017

Preprint

135

View full text Add to dashboard Cite

show abstract

“…Each threshold is associated with a value of precision and recall, corresponding to a point in the PR space. To obtain a single performance measure from the curve, we calculate the area under the curve using a trapezoidal approximation (Boyd et al, 2013). A perfect classifier would have an 30 AUPRC of 1.0.…”

Section: Measures Estimating the Predictive Performance Of Hierarchicmentioning

confidence: 99%

New KEGG pathway-based interpretable features for classifying ageing-related mouse proteins

Fabris

Freitas

2016

Bioinformatics

View full text Add to dashboard Cite

Motivation: The incidence of ageing-related diseases has been constantly increasing in the last decades, raising the need for creating effective methods to analyze ageing-related protein data. These methods should have high predictive accuracy and be easily interpretable by ageing experts. 15 To enable this, one needs interpretable classification models (supervised machine learning) and features with rich biological meaning. In this paper we propose two interpretable feature types based on Kyoto Encyclopedia of Genes and Genomes (KEGG) pathways and compare them with traditional feature types in hierarchical classification (a more challenging classification task regarding predictive performance) and binary classification (a classification task producing easier to inter-20 pret classification models). As far as we know, this work is the first to: (i) explore the potential of the KEGG pathway data in the hierarchical classification setting, (i) use the graph structure of KEGG pathways to create a feature type that quantifies the influence of a current protein on another specific protein within a KEGG pathway graph and (iii) propose a method for interpreting the classification models induced using KEGG features. 25 Results: We performed tests measuring predictive accuracy considering hierarchical and binary class labels extracted from the Mouse Phenotype Ontology. One of the KEGG feature types leads to the highest predictive accuracy among five individual feature types across three hierarchical classification algorithms. Additionally, the combination of the two KEGG feature types proposed in this work results in one of the best predictive accuracies when using the binary class version of our 30 datasets, at the same time enabling the extraction of knowledge from ageing-related data using quantitative influence information.

show abstract

Area under the Precision-Recall Curve: Point Estimates and Confidence Intervals

Cited by 317 publications

References 19 publications

Tversky Loss Function for Image Segmentation Using 3D Fully Convolutional Deep Networks

Tversky Loss Function for Image Segmentation Using 3D Fully Convolutional Deep Networks

Comprehensive Benchmarking and Ensemble Approaches for Metagenomic Classifiers

New KEGG pathway-based interpretable features for classifying ageing-related mouse proteins

Contact Info

Product

Resources

About