Data-Driven Rank Aggregation with Application to Grand Challenges

Fishbaugh, James; Prastawa, Marcel; Wang, Bo; Reynolds, Patrick; Aylward, Stephen; Gerig, Guido

doi:10.1007/978-3-319-66185-8_85

Cited by 3 publications

(4 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Limitations of this study included dataset bias (not all diseases, ages, devices, or ethnicities were represented equally across categories), and incomplete dermoscopic feature annotations. Reliance on single evaluation metrics rather than combinations may also be a limitation [27]. Future challenges will attempt to address these issues in conjunction with the community.…”

Section: Discussionmentioning

confidence: 99%

Skin lesion analysis toward melanoma detection: A challenge at the 2017 International symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC)

Codella

Gutman

Celebi

et al. 2018

2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018)

1,563

890

View full text Add to dashboard Cite

This work summarizes the results of the largest skin image analysis challenge in the world, hosted by the International Skin Imaging Collaboration (ISIC), a global partnership that has organized the world's largest public repository of dermoscopic images of skin. The challenge was hosted in 2018 at the Medical Image Computing and Computer Assisted Intervention (MICCAI) conference in Granada, Spain. The dataset included over 12,500 images across 3 tasks. 900 users registered for data download, 115 submitted to the lesion segmentation task, 25 submitted to the lesion attribute detection task, and 159 submitted to the disease classification task. Novel evaluation protocols were established, including a new test for segmentation algorithm performance, and a test for algorithm ability to generalize. Results show that top segmentation algorithms still fail on over 10% of images on average, and algorithms with equal performance on test data can have different abilities to generalize. This is an important consideration for agencies regulating the growing set of machine learning tools in the healthcare domain, and sets a new standard for future public challenges in healthcare.

show abstract

Section: Discussionmentioning

confidence: 99%

Skin lesion analysis toward melanoma detection: A challenge at the 2017 International symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC)

Codella

Gutman

Celebi

et al. 2018

2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018)

1,563

890

View full text Add to dashboard Cite

show abstract

“…However, it is important to point out that the top 5 teams outperformed other teams with a clear margin (average ≥6% in dice, recall and F-1 scores), while they themselves perform within marginal difference (Table 1). The specific ranking place may not be that important in this situation, as the commonly used average or accumulated ranking scheme may not reflect the importance of difference metrics for specific problems; hence, a different weighting scheme can dramatically change the ranking on real anatomical data [11].…”

Section: Experiments and Resultsmentioning

confidence: 99%

“…In this paper, we present a customized U-Net FCN with residual connections, namely ResU-Net, to segment WHM by using combined T1 and FLAIR images. Given the high sensitivity of challenge rankings to metric weights [11], we put less emphasis on achieving the highest rank in the WMH Segmentation Challenge. Instead, our goal is to perform comparably to the Challenge leaders while demonstrating excellent generalization ability, thereby providing confidence that our solution can be effectively used outside the confines of the challenges.…”

Section: Introductionmentioning

confidence: 99%

White matter hyperintensity segmentation from T1 and FLAIR images using fully convolutional neural networks enhanced with residual connections

Jin

Harrison

et al. 2018

2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018)

View full text Add to dashboard Cite

Segmentation and quantification of white matter hyperintensities (WMHs) are of great importance in studying and understanding various neurological and geriatric disorders. Although automatic methods have been proposed for WMH segmentation on magnetic resonance imaging (MRI), manual corrections are often necessary to achieve clinically practical results. Major challenges for WMH segmentation stem from their inhomogeneous MRI intensities, random location and size distributions, and MRI noise. The presence of other brain anatomies or diseases with enhanced intensities adds further difficulties. To cope with these challenges, we present a specifically designed fully convolutional neural network (FCN) with residual connections to segment WMHs by using combined T1 and fluid-attenuated inversion recovery (FLAIR) images. Our customized FCN is designed to be straightforward and generalizable, providing efficient end-to-end training due to its enhanced information propagation. We tested our method on the open WMH Segmentation Challenge MICCAI2017 dataset, and, despite our method's relative simplicity, results show that it performs amongst the leading techniques across five metrics. More importantly, our method achieves the best score for hausdorff distance and average volume difference in testing datasets from two MRI scanners that were not included in training, demonstrating better generalization ability of our proposed method over its competitors.Index Termswhite matter hyperintensity, MRI, fully convolutional neural network, residual connection Corresponding author: ziyue.xu@nih.gov.

show abstract

“…In addition to a challenge's data, truth, and metrics defining its type, they also impact its "efficacy". Others have covered the difficulties associated with selecting data [12], truth [15], and metrics [13] to create "effective" challenges; however, much of that work has focused on maximizing a studys statistical significance. There are three difficulties with focusing on statistical significance when optimizing a challenge: 1) statistical significance is associated with testing a hypothesis and therefore is only relevant to quantitative experiments; 2) the complex interrelationship between data, metrics, and truth is oversimplified by commonly used measures of statistical significance; and 3) statistical significance is not a measure of practical significance or generalizability.…”

Section: Insight Designmentioning

confidence: 99%

A Framework for Challenge Design: Insight and Deployment Challenges to Address Medical Image Analysis Problems

Mendrik¹,

Aylward²

2019

Preprint

View full text Add to dashboard Cite

In this paper we aim to refine the concept of grand challenges in medical image analysis, based on statistical principles from quantitative and qualitative experimental research. We identify two types of challenges based on their generalization objective: 1) a deployment challenge and 2) an insight challenge. A deployment challenge's generalization objective is to find algorithms that solve a medical image analysis problem, which thereby requires the use of a quantitative experimental design. An insight challenge's generalization objective is to gain a broad understanding of what class of algorithms might be effective for a class of medical image analysis problems, in which case a qualitative experimental design is sufficient. Both challenge types are valuable, but problems arise when a challenge's design and objective are inconsistent, as is often the case when a challenge does not carefully consider these concepts. Therefore, in this paper, we propose a theoretical framework, based on statistical principles, to guide researchers in challenge design, documentation, and assessment. Experimental results are given that explore the factors that effect the practical implementation of this theoretical framework.

show abstract

Data-Driven Rank Aggregation with Application to Grand Challenges

Cited by 3 publications

References 11 publications

Skin lesion analysis toward melanoma detection: A challenge at the 2017 International symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC)

Skin lesion analysis toward melanoma detection: A challenge at the 2017 International symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC)

White matter hyperintensity segmentation from T1 and FLAIR images using fully convolutional neural networks enhanced with residual connections

A Framework for Challenge Design: Insight and Deployment Challenges to Address Medical Image Analysis Problems

Contact Info

Product

Resources

About