No-gold-standard evaluation of quantitative imaging methods in the presence of correlated noise

Liu, Ziping; Li, Zekun; Mhlanga, Joyce; Siegel, Barry A.; Jha, Abhinav K.

doi:10.1117/12.2605762

Cited by 6 publications

(8 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To address this limitation, no-gold-standard evaluation techniques have been developed. [26][27][28][29] These techniques have demonstrated the efficacy in evaluating PET segmentation methods on clinically relevant tasks even without any knowledge of the ground truth. [30][31][32] Thus, these techniques could provide a mechanism to perform objective task-based evaluation of segmentation methods in the absence of ground truth.…”

Section: Discussionmentioning

confidence: 99%

“…[26][27][28][29] These techniques have demonstrated the efficacy in evaluating PET segmentation methods on clinically relevant tasks even without any knowledge of the ground truth. [30][31][32] Thus, these techniques could provide a mechanism to perform objective task-based evaluation of segmentation methods in the absence of ground truth. Another limitation of this study is that we evaluated the considered AI-based segmentation method only on the clinical tasks of quantifying MTV/TLG.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Need for objective task-based evaluation of AI-based segmentation methods for quantitative PET

Liu

Mhlanga

Siegel

et al. 2023

Medical Imaging 2023: Image Perception, Observer Performance, and Technology Assessment

View full text Add to dashboard Cite

Artificial intelligence (AI)-based methods are showing substantial promise in segmenting oncologic positron emission tomography (PET) images. For clinical translation of these methods, assessing their performance on clinically relevant tasks is important. However, these methods are typically evaluated using metrics that may not correlate with the task performance. One such widely used metric is the Dice score, a figure of merit that measures the spatial overlap between the estimated segmentation and a reference standard (e.g., manual segmentation). In this work, we investigated whether evaluating AI-based segmentation methods using Dice scores yields a similar interpretation as evaluation on the clinical tasks of quantifying metabolic tumor volume (MTV) and total lesion glycolysis (TLG) of primary tumor from PET images of patients with non-small cell lung cancer. The investigation was conducted via a retrospective analysis with the ECOG-ACRIN 6668/RTOG 0235 multi-center clinical trial data. Specifically, we evaluated different structures of a commonly used AI-based segmentation method using both Dice scores and the accuracy in quantifying MTV/TLG. Our results show that evaluation using Dice scores can lead to findings that are inconsistent with evaluation using the task-based figure of merit. Thus, our study motivates the need for objective task-based evaluation of AI-based segmentation methods for quantitative PET.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Need for objective task-based evaluation of AI-based segmentation methods for quantitative PET

Liu

Mhlanga

Siegel

et al. 2023

Medical Imaging 2023: Image Perception, Observer Performance, and Technology Assessment

View full text Add to dashboard Cite

show abstract

“…To address this challenge, no-gold-standard evaluation techniques have been developed. 51,62,[75][76][77][78] These techniques would provide a mechanism to evaluate the proposed method with patient data. A second limitation is that the proposed method focused on segmenting the caudate, putamen, and GP from DaT-SPECT images and considered the rest of the brain as background.…”

Section: Discussionmentioning

confidence: 99%

“…Validation with patient studies requires a gold standard that is not available in DaT‐SPECT studies. To address this challenge, no‐gold‐standard evaluation techniques have been developed 51,62,75–78 . These techniques would provide a mechanism to evaluate the proposed method with patient data.…”

Section: Discussionmentioning

confidence: 99%

A tissue‐fraction estimation‐based segmentation method for quantitative dopamine transporter SPECT

Liu

Moon

et al. 2022

Medical Physics

Self Cite

View full text Add to dashboard Cite

Background: Quantitative measures of dopamine transporter (DaT) uptake in caudate, putamen, and globus pallidus (GP) derived from dopamine transporter-single-photon emission computed tomography (DaT-SPECT) images have potential as biomarkers for measuring the severity of Parkinson's disease. Reliable quantification of this uptake requires accurate segmentation of the considered regions. However, segmentation of these regions from DaT-SPECT images is challenging, a major reason being partial-volume effects (PVEs) in SPECT. The PVEs arise from two sources, namely the limited system resolution and reconstruction of images over finite-sized voxel grids. The limited system resolution results in blurred boundaries of the different regions. The finite voxel size leads to TFEs, that is, voxels contain a mixture of regions. Thus, there is an important need for methods that can account for the PVEs, including the TFEs, and accurately segment the caudate, putamen, and GP, from DaT-SPECT images. Purpose: Design and objectively evaluate a fully automated tissue-fraction estimation-based segmentation method that segments the caudate, putamen, and GP from DaT-SPECT images. Methods: The proposed method estimates the posterior mean of the fractional volumes occupied by the caudate, putamen, and GP within each voxel of a three-dimensional DaT-SPECT image. The estimate is obtained by minimizing a cost function based on the binary cross-entropy loss between the true and estimated fractional volumes over a population of SPECT images, where the distribution of true fractional volumes is obtained from existing populations of clinical magnetic resonance images. The method is implemented using a supervised deep-learning-based approach. Results: Evaluations using clinically guided highly realistic simulation studies show that the proposed method accurately segmented the caudate, putamen, and GP with high mean Dice similarity coefficients of ∼ 0.80 and significantly outperformed (p < 0.01) all other considered segmentation methods. Further, an objective evaluation of the proposed method on the task of quantifying regional uptake shows that the method yielded reliable quantification with low ensemble normalized root mean square error (NRMSE) < 20% for all the considered regions. In particular, the method yielded an even lower ensemble NRMSE of ∼ 10% for the caudate and putamen. Conclusions: The proposed tissue-fraction estimation-based segmentation method for DaT-SPECT images demonstrated the ability to accurately segment the caudate, putamen, and GP, and reliably quantify the uptake within these regions. The results motivate further evaluation of the method with physical-phantom and patient studies.

show abstract

“…Validation with patient studies requires a gold standard and that is not available for these types of images. To address this challenge, no-gold-standard (NGS) evaluation techniques have been developed (Kupinski et al, 2002;Jha et al, 2012Jha et al, , 2016Jha et al, , 2017bLiu et al, 2020). These techniques would provide an approach to evaluate the proposed method with patient studies.…”

Section: Discussionmentioning

confidence: 99%

A tissue-fraction estimation-based segmentation method for quantitative dopamine transporter SPECT

Liu,

Moon,

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Quantitative measures of uptake in the caudate, putamen, and globus pallidus in dopamine transporter (DaT) brain SPECT have potential as biomarkers for the severity of Parkinson disease. This potential is critical for clinical care and research. Reliable quantification of uptake requires accurate segmentation of these regions. However, segmentation is challenging in DaT SPECT due to partial-volume effects, system noise, physiological variability, and the small size of these regions. To address these challenges, we propose an estimation-based approach to segmentation. This approach estimates the posterior mean of the fractional volume occupied by caudate, putamen, and globus pallidus within each voxel of a 3D SPECT image. The estimate is obtained by minimizing a cost function based on the binary cross-entropy loss between the true and estimated fractional volumes over a population of SPECT images, where the distribution of the true fractional volumes is obtained from magnetic resonance (MR) images from clinical populations. The proposed method accounts for both the sources of partial-volume effects in SPECT, namely the limited system resolution and tissue-fraction effects. The method was implemented using an encoder-decoder network and evaluated using realistic clinically guided SPECT simulation studies, where the ground-truth fractional volumes of caudate, putamen, and globus pallidus were known. The method significantly outperformed all other considered segmentation methods, including a U-net-based method, and yielded accurate segmentation with dice similarity coefficients of ∼ 0.80 for all regions. The method was relatively insensitive to the changes in voxel size. Further, the method was relatively robust up to ±10 • of patient head tilt along transaxial, sagittal, and coronal planes. Overall, the results demonstrate the efficacy of the proposed method to yield accurate fully automated segmentation of caudate, putamen, and globus pallidus in 3D DaT-SPECT images.

show abstract

No-gold-standard evaluation of quantitative imaging methods in the presence of correlated noise

Cited by 6 publications

References 24 publications

Need for objective task-based evaluation of AI-based segmentation methods for quantitative PET

Need for objective task-based evaluation of AI-based segmentation methods for quantitative PET

A tissue‐fraction estimation‐based segmentation method for quantitative dopamine transporter SPECT

A tissue-fraction estimation-based segmentation method for quantitative dopamine transporter SPECT

Contact Info

Product

Resources

About