Anatomical and Diagnostic Bayesian Segmentation in Prostate MRI $-$Should Different Clinical Objectives Mandate Different Loss Functions?

Saha, Anindo; S., Bosma, Joeran; Linmans, Jasper; Hosseinzadeh, Matin; Huisman, Henkjan

doi:10.48550/arxiv.2110.12889

Cited by 3 publications

(3 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…clinical) validation goal. Previous research, however, suggests that this is often not the case [73]. Before choosing validation metrics, the correct problem category needs to be defined.…”

Section: D1 Pitfalls Related To An Inadequate Choice Of the Problem C...mentioning

confidence: 99%

Understanding metric-related pitfalls in image analysis validation

Reinke¹,

Tizabi²,

Baumgärtner³

et al. 2023

Preprint

View full text Add to dashboard Cite

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

show abstract

“…clinical) validation goal. Previous research, however, suggests that this is often not the case [73]. Before choosing validation metrics, the correct problem category needs to be defined.…”

Section: D1 Pitfalls Related To An Inadequate Choice Of the Problem C...mentioning

confidence: 99%

Understanding metric-related pitfalls in image analysis validation

Reinke¹,

Tizabi²,

Baumgärtner³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Focal loss was used to counteract bias due to class imbalance in the dataset, and to implicitly calibrate the model at train-time [17][18][19]. This configuration has previously been shown to outperform similar models for the detection of PCa on bpMRI [20,21].…”

Section: Mri Surveillance Featuresmentioning

confidence: 99%

AI-assisted biparametric MRI surveillance of prostate cancer: feasibility study

et al. 2022

View full text Add to dashboard Cite

Objectives To evaluate the feasibility of automatic longitudinal analysis of consecutive biparametric MRI (bpMRI) scans to detect clinically significant (cs) prostate cancer (PCa). Methods This retrospective study included a multi-center dataset of 1513 patients who underwent bpMRI (T2 + DWI) between 2014 and 2020, of whom 73 patients underwent at least two consecutive bpMRI scans and repeat biopsies. A deep learning PCa detection model was developed to produce a heatmap of all PIRADS ≥ 2 lesions across prior and current studies. The heatmaps for each patient’s prior and current examination were used to extract differential volumetric and likelihood features reflecting explainable changes between examinations. A machine learning classifier was trained to predict from these features csPCa (ISUP > 1) at the current examination according to biopsy. A classifier trained on the current study only was developed for comparison. An extended classifier was developed to incorporate clinical parameters (PSA, PSA density, and age). The cross-validated diagnostic accuracies were compared using ROC analysis. The diagnostic performance of the best model was compared to the radiologist scores. Results The model including prior and current study (AUC 0.81, CI: 0.69, 0.91) resulted in a higher (p = 0.04) diagnostic accuracy than the current only model (AUC 0.73, CI: 0.61, 0.84). Adding clinical variables further improved diagnostic performance (AUC 0.86, CI: 0.77, 0.93). The diagnostic performance of the surveillance AI model was significantly better (p = 0.02) than of radiologists (AUC 0.69, CI: 0.54, 0.81). Conclusions Our proposed AI-assisted surveillance of prostate MRI can pick up explainable, diagnostically relevant changes with promising diagnostic accuracy. Key Points • Sequential prostate MRI scans can be automatically evaluated using a hybrid deep learning and machine learning approach. • The diagnostic accuracy of our csPCa detection AI model improved by including clinical parameters.

show abstract

“…clinical goal). Previous research, however, suggests, that this is often not the case [78]. Before choosing validation metrics, the correct problem category needs to be defined.…”

Section: H Pitfalls Of Metricsmentioning

confidence: 99%

Metrics reloaded: Pitfalls and recommendations for image analysis validation

Maier-Hein¹,

Reinke²,

Godau³

et al. 2022

Preprint

View full text Add to dashboard Cite

The field of automatic biomedical image analysis crucially depends on robust and meaningful performance metrics for algorithm validation. Current metric usage, however, is often ill-informed and does not reflect the underlying domain interest. Here, we present a comprehensive framework that guides researchers towards choosing performance metrics in a problem-aware manner. Specifically, we focus on biomedical image analysis problems that can be interpreted as a classification task at image, object or pixel level. The framework first compiles domain interest-, target structure-, data set-and algorithm output-related properties of a given problem into a problem fingerprint, while also mapping it to the appropriate problem category, namely image-level classification, semantic segmentation, instance segmentation, or object detection. It then guides users through the process of selecting and applying a set of appropriate validation metrics while making them aware of potential pitfalls related to individual choices. In this paper, we describe the current status of the Metrics Reloaded recommendation framework, with the goal of obtaining constructive feedback from the image analysis community. The current version has been developed within an international consortium of more than 60 image analysis experts and will be made openly available as a user-friendly toolkit after community-driven optimization.

show abstract

Anatomical and Diagnostic Bayesian Segmentation in Prostate MRI $-$Should Different Clinical Objectives Mandate Different Loss Functions?

Cited by 3 publications

References 29 publications

Understanding metric-related pitfalls in image analysis validation

Understanding metric-related pitfalls in image analysis validation

AI-assisted biparametric MRI surveillance of prostate cancer: feasibility study

Metrics reloaded: Pitfalls and recommendations for image analysis validation

Contact Info

Product

Resources

About