Evaluating participating methods in image analysis challenges: Lessons from MoNuSAC 2020

Foucart, Adrien; Debeir, Olivier; Decaestecker, Christine

doi:10.1016/j.patcog.2023.109600

Cited by 3 publications

(3 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Then, if a single final ranking is needed, a method like the sum of ranks used in the GlaS 2015 challenge 16 can be used. We previously demonstrated on the MoNuSAC challenge how such a separation could lead to more informative results 11 .…”

Section: Discussionmentioning

confidence: 99%

“…PQ was then adopted as the ranked metric of the MoNuSAC 2020 challenge 3 , then the CoNIC 2022 challenge 4 , and was used in several recent publications as a means of comparison with the state-of-the-art 5 – 9 and/or to make choices in algorithm development 10 . Yet, as we show in our analysis of the MoNuSAC results 11 , this metric can hide much useful information about the performance of competing algorithms. In this work, we analyse more generally why the PQ metric is not suitable for cell nucleus instance segmentation and classification and should therefore be avoided.…”

Section: Introductionmentioning

confidence: 94%

“…As we have shown in a previous work 11 , the PQ metric hides a lot of potentially insightful information about the performances of algorithms by merging information of a very different nature. While SQ and RQ have the same range of possible values, being bounded between 0 and 1, multiplying these values to get PQ implies that the impact of a change in SQ by a factor a is exactly the same as a change of RQ by the same factor.…”

Section: Theoretical Analysismentioning

confidence: 99%

See 2 more Smart Citations

Panoptic quality should be avoided as a metric for assessing cell nuclei segmentation and classification in digital pathology

Foucart

Debeir

Decaestecker

2023

Sci Rep

Self Cite

View full text Add to dashboard Cite

Panoptic Quality (PQ), designed for the task of “Panoptic Segmentation” (PS), has been used in several digital pathology challenges and publications on cell nucleus instance segmentation and classification (ISC) since its introduction in 2019. Its purpose is to encompass the detection and the segmentation aspects of the task in a single measure, so that algorithms can be ranked according to their overall performance. A careful analysis of the properties of the metric, its application to ISC and the characteristics of nucleus ISC datasets, shows that is not suitable for this purpose and should be avoided. Through a theoretical analysis we demonstrate that PS and ISC, despite their similarities, have some fundamental differences that make PQ unsuitable. We also show that the use of the Intersection over Union as a matching rule and as a segmentation quality measure within PQ is not adapted for such small objects as nuclei. We illustrate these findings with examples taken from the NuCLS and MoNuSAC datasets. The code for replicating our results is available on GitHub (https://github.com/adfoucart/panoptic-quality-suppl).

show abstract

Section: Discussionmentioning

confidence: 99%