Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficient

Kofler, Florian; Ezhov, Ivan; Isensee, Fabian; Balsiger, Fabian; Berger, Christoph; Koerner, Maximilian; Demiray, Beatrice; Rackerseder, Julia; Paetzold, Johannes C.; Li, Hongwei; Shit, Suprosanna; McKinley, Richard L.; Piraud, Marie; Bakas, Spyridon; Zimmer, Claus; Navab, Nassir; Kirschke, Jan S.; Wiestler, Benedikt; Menze, Bjoern

doi:10.48550/arxiv.2103.06205

Cited by 17 publications

(25 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The global Dice scores for the different classes were strongly influenced by the prevalence of each class in the data. The classes that included many voxels and were present in most or all slices had high global Dice values, whereas the classes that were only present in a few slices and were completely absent for some scans had low global Dice values [ 36 ]. This is a well-known limitation of the Dice metric [ 37 ] and is particularly evident for our application where there is also a high degree of uncertainty in the precise voxel-wise labelling of the classes, especially in the hold out test dataset.…”

Section: Discussionmentioning

confidence: 99%

A Novel and Automated Approach to Classify Radiation Induced Lung Tissue Damage on CT Scans

Szmul

Chandy

Veiga

et al. 2022

Cancers

View full text Add to dashboard Cite

Radiation-induced lung damage (RILD) is a common side effect of radiotherapy (RT). The ability to automatically segment, classify, and quantify different types of lung parenchymal change is essential to uncover underlying patterns of RILD and their evolution over time. A RILD dedicated tissue classification system was developed to describe lung parenchymal tissue changes on a voxel-wise level. The classification system was automated for segmentation of five lung tissue classes on computed tomography (CT) scans that described incrementally increasing tissue density, ranging from normal lung (Class 1) to consolidation (Class 5). For ground truth data generation, we employed a two-stage data annotation approach, akin to active learning. Manual segmentation was used to train a stage one auto-segmentation method. These results were manually refined and used to train the stage two auto-segmentation algorithm. The stage two auto-segmentation algorithm was an ensemble of six 2D Unets using different loss functions and numbers of input channels. The development dataset used in this study consisted of 40 cases, each with a pre-radiotherapy, 3-, 6-, 12-, and 24-month follow-up CT scans (n = 200 CT scans). The method was assessed on a hold-out test dataset of 6 cases (n = 30 CT scans). The global Dice score coefficients (DSC) achieved for each tissue class were: Class (1) 99% and 98%, Class (2) 71% and 44%, Class (3) 56% and 26%, Class (4) 79% and 47%, and Class (5) 96% and 92%, for development and test subsets, respectively. The lowest values for the test subsets were caused by imaging artefacts or reflected subgroups that occurred infrequently and with smaller overall parenchymal volumes. We performed qualitative evaluation on the test dataset presenting manual and auto-segmentation to a blinded independent radiologist to rate them as ‘acceptable’, ‘minor disagreement’ or ‘major disagreement’. The auto-segmentation ratings were similar to the manual segmentation, both having approximately 90% of cases rated as acceptable. The proposed framework for auto-segmentation of different lung tissue classes produces acceptable results in the majority of cases and has the potential to facilitate future large studies of RILD.

show abstract

Section: Discussionmentioning

confidence: 99%

A Novel and Automated Approach to Classify Radiation Induced Lung Tissue Damage on CT Scans

Szmul

Chandy

Veiga

et al. 2022

Cancers

View full text Add to dashboard Cite

show abstract

“…Therefore, a hypothetical perfect segmentation made by an algorithm (extraction of the whole tumor and no non-tumoral pixels) would not obtain a maximum score since the metric measures the difference between the contours drawn by humans and those drawn by the algorithm. Furthermore, a weak correlation between expert assessment and traditional metrics has been noted 27 . We used the DICE coefficient, the most used metric in biomedical image analysis competition 28 , even though there is no consensus on the method to evaluate the models.…”

Section: Discussionmentioning

confidence: 99%

A deep learning model trained on only eight whole-slide images accurately segments tumors: wise data use versus big data

Perennec

Bourgade

Henno

et al. 2022

Preprint

View full text Add to dashboard Cite

Computer-assisted pathology is one of the biggest challenges in the medicine of the future. However, artificial intelligence is struggling to gain acceptance in the broader medical community due to data security issues, lack of trust in the machine, and poor data availability. Here, we develop a tumor delineation algorithm with only eight whole slide images of ovarian cancer to demonstrate the feasibility of an artificial intelligence application created from only a few data, finely annotated and with optimal processing. We test the model on seventeen other slides from the same hospital. The predictions are similar to the ground truth annotations made by an expert pathologist, with a mean DICE score of 0.90 [0.85 - 0.93]. The results on slides from another hospital are consistent, suggesting that the model is generalizable and that its performance does not suffer from different data acquisition. This study demonstrates the feasibility of a contouring algorithm based on a reduced dataset well optimized, going against the commonly accepted idea that a phenomenal amount of data is paramount. This study paves the way for other medical applications, especially for rare pathologies with limited available data.

show abstract

“…The Dice score and the Hausdorff distance are the two most standard metrics used for measuring the quality of automatic segmentations [4]. However, those two metrics do not directly measure the trustworthiness of segmentation algorithms [33]. Therefore, we have also conducted an evaluation of the trustworthiness of the automatic segmentations as perceived by radiologists.…”

Section: Stratified Evaluation Across Brain Conditions and Acquisitio...mentioning

confidence: 99%

A Dempster-Shafer approach to trustworthy AI with application to fetal brain MRI segmentation

Fidon¹,

Aertsen²,

Wiestler³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficient

Cited by 17 publications

References 28 publications

A Novel and Automated Approach to Classify Radiation Induced Lung Tissue Damage on CT Scans

A Novel and Automated Approach to Classify Radiation Induced Lung Tissue Damage on CT Scans

A deep learning model trained on only eight whole-slide images accurately segments tumors: wise data use versus big data

A Dempster-Shafer approach to trustworthy AI with application to fetal brain MRI segmentation

Contact Info

Product

Resources

About