Objective Task-Based Evaluation of Artificial Intelligence-Based Medical Imaging Methods

Jha, Abhinav K.; Myers, Kyle J.; Obuchowski, Nancy A.; Liu, Ziping; Rahman, Ashequr; Saboury, Babak; Rahmim, Arman; Siegel, Barry A.

doi:10.1016/j.cpet.2021.06.013

Cited by 35 publications

(42 citation statements)

References 91 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[25][26][27] Thus, these methods should preferably be evaluated based on clinical-task performance. 28 The proposed NGSE technique provides a mechanism to perform evaluation on clinically relevant quantitative tasks and without access to the ground truth.…”

Section: Discussionmentioning

confidence: 99%

No-gold-standard evaluation of quantitative imaging methods in the presence of correlated noise

Liu

Mhlanga³

et al. 2022

Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment

Self Cite

View full text Add to dashboard Cite

Objective evaluation of quantitative imaging (QI) methods with patient data is highly desirable, but is hindered by the lack or unreliability of an available gold standard. To address this issue, techniques that can evaluate QI methods without access to a gold standard are being actively developed. These techniques assume that the true and measured values are linearly related by a slope, bias, and Gaussian-distributed noise term, where the noise between measurements made by different methods is independent of each other. However, this noise arises in the process of measuring the same quantitative value, and thus can be correlated. To address this limitation, we propose a no-gold-standard evaluation (NGSE) technique that models this correlated noise by a multi-variate Gaussian distribution parameterized by a covariance matrix. We derive a maximum-likelihood-based approach to estimate the parameters that describe the relationship between the true and measured values, without any knowledge of the true values. We then use the estimated slopes and diagonal elements of the covariance matrix to compute the noise-to-slope ratio (NSR) to rank the QI methods on the basis of precision. The proposed NGSE technique was evaluated with multiple numerical experiments. Our results showed that the technique reliably estimated the NSR values and yielded accurate rankings of the considered methods for 83% of 160 trials. In particular, the technique correctly identified the most precise method for ∼ 97% of the trials. Overall, this study demonstrates the efficacy of the NGSE technique to accurately rank different QI methods when the correlated noise is present, and without access to any knowledge of the ground truth. The results motivate further validation of this technique with realistic simulation studies and patient data.

show abstract

Section: Discussionmentioning

confidence: 99%

No-gold-standard evaluation of quantitative imaging methods in the presence of correlated noise

Liu

Mhlanga³

et al. 2022

Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment

Self Cite

View full text Add to dashboard Cite

show abstract

“…To perform such objective task-based evaluation of DL-based methods, a framework was recently outlined. 26 Further, best practices are being defined to objectively evaluate AI-based methods for nuclear medicine. 27 Moreover, recent studies are investigating the evaluation of DL-based imaging methods for nuclear medicine on specific clinical tasks.…”

Section: Discussionmentioning

confidence: 99%

Investigating the limited performance of a deep-learning-based SPECT denoising approach: an observer-study-based characterization

Rahman

Jha

2022

Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment

Self Cite

View full text Add to dashboard Cite

Multiple objective assessment of image-quality (OAIQ)-based studies have reported that several deep-learning (DL)-based denoising methods show limited performance on signal-detection tasks. Our goal was to investigate the reasons for this limited performance. To achieve this goal, we conducted a task-based characterization of a DL-based denoising approach for individual signal properties. We conducted this study in the context of evaluating a DL-based approach for denoising single photon-emission computed tomography (SPECT) images.The training data consisted of signals of different sizes and shapes within a clustered-lumpy background, imaged with a 2D parallel-hole-collimator SPECT system. The projections were generated at normal and 20% low-count level, both of which were reconstructed using an ordered-subset-expectation-maximization (OSEM) algorithm. A convolutional neural network (CNN)-based denoiser was trained to process the low-count images. The performance of this CNN was characterized for five different signal sizes and four different signal-tobackground ratio (SBRs) by designing each evaluation as a signal-known-exactly/background-known-statistically (SKE/BKS) signal-detection task. Performance on this task was evaluated using an anthropomorphic channelized Hotelling observer (CHO). As in previous studies, we observed that the DL-based denoising method did not improve performance on signal-detection tasks. Evaluation using the idea of observer-study-based characterization demonstrated that the DL-based denoising approach did not improve performance on the signal-detection task for any of the signal types. Overall, these results provide new insights on the performance of the DL-based denoising approach as a function of signal size and contrast. More generally, the observer study-based characterization provides a mechanism to evaluate the sensitivity of the method to specific object properties, and may be explored as analogous to characterizations such as modulation transfer function for linear systems. Finally, this work underscores the need for objective task-based evaluation of DL-based denoising approaches.

show abstract

“…18 Lastly, we emphasize that a task-based assessment of image reconstruction methods is needed in order to translate these methods into the clinic. 1,19,20…”

Section: Discussionmentioning

confidence: 99%

Prior image-based medical image reconstruction using a style-based generative adversarial network

Kelkar¹,

Anastasio²

2022

Preprint

View full text Add to dashboard Cite

Computed medical imaging systems require a computational reconstruction procedure for image formation. In order to recover a useful estimate of the object to-be-imaged when the recorded measurements are incomplete, prior knowledge about the nature of object must be utilized. In order to improve the conditioning of an ill-posed imaging inverse problem, deep learning approaches are being actively investigated for better representing object priors and constraints. This work proposes to use a style-based generative adversarial network (StyleGAN) to constrain an image reconstruction problem in the case where additional information in the form of a prior image of the sought-after object is available. An optimization problem is formulated in the intermediate latent-space of a StyleGAN, that is disentangled with respect to meaningful image attributes or "styles", such as the contrast used in magnetic resonance imaging (MRI). Discrepancy between the sought-after and prior images is measured in the disentangled latent-space, and is used to regularize the inverse problem in the form of constraints on specific styles of the disentangled latent-space. A stylized numerical study inspired by MR imaging is designed, where the sought-after and the prior image are structurally similar, but belong to different contrast mechanisms. The presented numerical studies demonstrate the superiority of the proposed approach as compared to classical approaches in the form of traditional metrics.

show abstract

Objective Task-Based Evaluation of Artificial Intelligence-Based Medical Imaging Methods

Cited by 35 publications

References 91 publications

No-gold-standard evaluation of quantitative imaging methods in the presence of correlated noise

No-gold-standard evaluation of quantitative imaging methods in the presence of correlated noise

Investigating the limited performance of a deep-learning-based SPECT denoising approach: an observer-study-based characterization

Prior image-based medical image reconstruction using a style-based generative adversarial network

Contact Info

Product

Resources

About