Machine-learning denoising in feature film production

Dahlberg, Henrik; Adler, David A.; Newlin, Jeremy

doi:10.1145/3306307.3328150

Cited by 8 publications

(5 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These techniques are designed for completely unstructured data and potentially require significant adaptation and architectural redesign to fit our specific denoising problem. Given the proven success of CNNs in denoising structured flat images in production [DAN19, ZZR*23], we opted to use a CNN denoiser for the deep‐Z images. Our work leverages the structure of deep‐Z images, particularly the 2‐D uniform pixel grid, and introduces depth‐aware neighborhoods to effectively denoise deep‐Z images, even with irregular depth structures.…”

Section: Background and Related Workmentioning

confidence: 99%

Neural Denoising for Deep‐Z Monte Carlo Renderings

Zhang,

Röthlin,

Zhu

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

We present a kernel‐predicting neural denoising method for path‐traced deep‐Z images that facilitates their usage in animation and visual effects production. Deep‐Z images provide enhanced flexibility during compositing as they contain color, opacity, and other rendered data at multiple depth‐resolved bins within each pixel. However, they are subject to noise, and rendering until convergence is prohibitively expensive. The current state of the art in deep‐Z denoising yields objectionable artifacts, and current neural denoising methods are incapable of handling the variable number of depth bins in deep‐Z images. Our method extends kernel‐predicting convolutional neural networks to address the challenges stemming from denoising deep‐Z images. We propose a hybrid reconstruction architecture that combines the depth‐resolved reconstruction at each bin with the flattened reconstruction at the pixel level. Moreover, we propose depth‐aware neighbor indexing of the depth‐resolved inputs to the convolution and denoising kernel application operators, which reduces artifacts caused by depth misalignment present in deep‐Z images. We evaluate our method on a production‐quality deep‐Z dataset, demonstrating significant improvements in denoising quality and performance compared to the current state‐of‐the‐art deep‐Z denoiser. By addressing the significant challenge of the cost associated with rendering path‐traced deep‐Z images, we believe that our approach will pave the way for broader adoption of deep‐Z workflows in future productions.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Neural Denoising for Deep‐Z Monte Carlo Renderings

Zhang,

Röthlin,

Zhu

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…They also decomposed the denoising pipeline with task-specific modules, independently extracting the source-aware and spatio-temporal information. Dahlberg et al [DAN19] deployed this modular hierarchical kernel prediction method to the existing commercial Monte Carlo path tracing engines and showed its practical ability and flexibility in denoising the feature film productions. However, the original kernel prediction method is designed mainly for offline applications with 16-64 spp and runs in seconds.…”

Section: Related Workmentioning

confidence: 99%

Real-time Monte Carlo Denoising with Weight Sharing Kernel Prediction Network

Fan,

Wang,

Huo

et al. 2022

Preprint

View full text Add to dashboard Cite

Real-time Monte Carlo denoising aims at removing severe noise under low samples per pixel (spp) in a strict time budget. Recently, kernel-prediction methods use a neural network to predict each pixel's filtering kernel and have shown a great potential to remove Monte Carlo noise. However, the heavy computation overhead blocks these methods from real-time applications. This paper expands the kernel-prediction method and proposes a novel approach to denoise very low spp (e.g., 1-spp) Monte Carlo path traced images at real-time frame rates. Instead of using the neural network to directly predict the kernel map, i.e., the complete weights of each per-pixel filtering kernel, we predict an encoding of the kernel map, followed by a high-efficiency decoder with unfolding operations for a high-quality reconstruction of the filtering kernels . The kernel map encoding yields a compact single-channel representation of the kernel map, which can significantly reduce the kernel-prediction network's throughput. In addition, we adopt a scalable kernel fusion module to improve denoising quality. The proposed approach preserves kernel prediction methods' denoising quality while roughly halving its denoising time for 1-spp noisy inputs. In addition, compared with the recent neural bilateral grid-based real-time denoiser, our approach benefits from the high parallelism of kernel-based reconstruction and produces better denoising results at equal time.

show abstract

“…Rendering realistic images for virtual worlds is a key objective in many computer vision and graphics tasks (Huo and Yoon 2021;Xu et al 2022;Huang et al 2023;Li, Ngo, and Nagahara 2023), with applications in animation production (Dahlberg, Adler, and Newlin 2019), VR/AR world generation (Overbeck et al 2018), virtual dataset synthesis (Ge et al 2022), etc. One widely used technique for this purpose is Monte Carlo (MC) sampling (Seila 1982), which is highly versatile but typically requires a large number of samples to achieve accurate results.…”

Section: Introductionmentioning

confidence: 99%

High-Quality Real-Time Rendering Using Subpixel Sampling Reconstruction

Zhang,

Yuan

2024

AAAI

View full text Add to dashboard Cite

Generating high-quality, realistic rendering images for real-time applications generally requires tracing a few samples-per-pixel (spp) and using deep learning-based approaches to denoise the resulting low-spp images. Existing denoising methods necessitate a substantial time expenditure when rendering at high resolutions due to the physically-based sampling and network inference time burdens. In this paper, we propose a novel Monte Carlo sampling strategy to accelerate the sampling process and a corresponding denoiser, subpixel sampling reconstruction (SSR), to obtain high-quality images. Extensive experiments demonstrate that our method significantly outperforms previous approaches in denoising quality and reduces overall time costs, enabling real-time rendering capabilities at 2K resolution.

show abstract

Machine-learning denoising in feature film production

Cited by 8 publications

References 3 publications

Neural Denoising for Deep‐Z Monte Carlo Renderings

Neural Denoising for Deep‐Z Monte Carlo Renderings

Real-time Monte Carlo Denoising with Weight Sharing Kernel Prediction Network

High-Quality Real-Time Rendering Using Subpixel Sampling Reconstruction

Contact Info

Product

Resources

About