Investigating the limited performance of a deep-learning-based SPECT denoising approach: an observer-study-based characterization

Abstract

Multiple objective assessment of image-quality (OAIQ)-based studies have reported that several deep-learning (DL)-based denoising methods show limited performance on signal-detection tasks. Our goal was to investigate the reasons for this limited performance. To achieve this goal, we conducted a task-based characterization of a DL-based denoising approach for individual signal properties. We conducted this study in the context of evaluating a DL-based approach for denoising single photon-emission computed tomography (SPECT) images.The training data consisted of signals of different sizes and shapes within a clustered-lumpy background, imaged with a 2D parallel-hole-collimator SPECT system. The projections were generated at normal and 20% low-count level, both of which were reconstructed using an ordered-subset-expectation-maximization (OSEM) algorithm. A convolutional neural network (CNN)-based denoiser was trained to process the low-count images. The performance of this CNN was characterized for five different signal sizes and four different signal-tobackground ratio (SBRs) by designing each evaluation as a signal-known-exactly/background-known-statistically (SKE/BKS) signal-detection task. Performance on this task was evaluated using an anthropomorphic channelized Hotelling observer (CHO). As in previous studies, we observed that the DL-based denoising method did not improve performance on signal-detection tasks. Evaluation using the idea of observer-study-based characterization demonstrated that the DL-based denoising approach did not improve performance on the signal-detection task for any of the signal types. Overall, these results provide new insights on the performance of the DL-based denoising approach as a function of signal size and contrast. More generally, the observer study-based characterization provides a mechanism to evaluate the sensitivity of the method to specific object properties, and may be explored as analogous to characterizations such as modulation transfer function for linear systems. Finally, this work underscores the need for objective task-based evaluation of DL-based denoising approaches.

Nuclear Medicine and Artificial Intelligence: Best Practices for Evaluation (the RELAINCE Guidelines)

“…For example, a technical efficacy study may observe suboptimal performance of an AI-based denoising algorithm on the tumor-detection task. Then, the evaluation study could investigate the performance of the algorithm for different tumor properties (size/tumor-to-background ratio) on the detection task (66). This will provide insights on the working principles of the algorithm, thus improving the interpretability of the algorithm.…”

Modeling human observer detection in undersampled magnetic resonance imaging reconstruction with total variation and wavelet sparsity regularization

“…22 Recent work studying the effect of denoising using neural networks on detection performance in single positron emission computed tomography showed that denoising could decrease detection performance even if it improved RMSE and SSIM. 30 In another study using simulated images showed similar results. 31 Exploring applications where regularization helps in a detection-based perspective would be useful to better understand the regimes where metrics like ERMSE and detection performance agree and disagree.…”

“…Some slight improvement was seen in the context of ramp-spectrum noise for human observers 17 and in ideal observer performance in undersampled MRI 22 . Recent work studying the effect of denoising using neural networks on detection performance in single positron emission computed tomography showed that denoising could decrease detection performance even if it improved RMSE and SSIM 30 . In another study using simulated images showed similar results 31 .…”

Need for objective task‐based evaluation of deep learning‐based denoising methods: A study in the context of myocardial perfusion SPECT

“…observed no significant correlation between peak signal‐to‐noise ratio and SSIM values and classification performance for a tumor classification task in chest radiographs 19 . A study by our group observed these limitations with a 2D single photon emission computed tomography (SPECT) system with lumpy background‐based tracer distribution models 21 . While these studies show the limitations of these FoMs, it is unclear if the results from these studies are indicators of performance in clinical settings in nuclear medicine.…”