Neural Denoising for Path Tracing of Medical Volumetric Data

Abstract

In this paper, we transfer machine learning techniques previously applied to denoising surface-only Monte Carlo renderings to path-traced visualizations of medical volumetric data. In the domain of medical imaging, path-traced videos turned out to be an efficient means to visualize and understand internal structures, in particular for less experienced viewers such as students or patients. However, the computational demands for the rendering of high-quality path-traced videos are very high due to the large number of samples necessary for each pixel. To accelerate the process, we present a learning-based technique for denoising path-traced videos of volumetric data by increasing the sample count per pixel; both through spatial (integrating neighboring samples) and temporal filtering (reusing samples over time). Our approach uses a set of additional features and a loss function both specifically designed for the volumetric case. Furthermore, we present a novel network architecture tailored for our purpose, and introduce reprojection of samples to improve temporal stability and reuse samples over frames. As a result, we achieve good image quality even from severely undersampled input images, as visible in the teaser image.

Automatic Feature Selection for Denoising Volumetric Renderings

“…There are also other volumetric effects such as volumetric emission ( e.g ., fire) and subsurface scattering ( e.g ., skin) [NGHJ18]. Denoisers for renderings of medical data have also been studied [HMES20]. Our method is not special to particular types of volumetric effects, and in principle can be used to discover good feature sets for denoising these effects as well.…”

“…Our work focuses on identifying auxiliary features sets for volume denoising rather than suitable volume denoising network architectures. Therefore, we examine the effect of our proposed feature set and the feature selection method on existing direct‐predicting [HMES20] and kernel‐predicting [HHCM21, VRM∗18] denoisers. Note that two of the three denoiser architectures [HMES20, HHCM21] are proposed in the context of volume denoising, with their own sets of volumetric features.…”

“…We provide feature sets obtained by our method for a selection of state‐of‐the‐art interactive and offline denoisers for volumes [VRM∗18, HMES20, HHCM21]. We demonstrate that these sets improve the denoising quality compared to the hand‐selected feature sets initially proposed by the authors of the respective methods.…”

“…Before the rendering converges to a noise‐free state, the process is stopped early, and a comparatively fast denoising post‐process can be applied to remove the noise in the intermediate image. State‐of‐the‐art denoisers [VRM∗18,CKS∗17,XZW∗19,ZMV∗21,HMES20,HHCM21] are based on training deep convolutional neural networks to produce noise‐free images from noisy inputs. While training denoising networks on large datasets can be very time consuming, once trained, these networks can be used to very efficiently produce clean images from noisy data.…”

“…In this paper, we propose an algorithm to identify sets of features that significantly improve the quality of various state‐of‐the‐art methods [VRM∗18, HMES20, HHCM21] that can be used for volume denoising. We progressively expand our selection of features from a more extensive hand‐crafted set to identify features that will yield the best improvement in the denoising quality.…”