Noise-based enhancement for foveated rendering

Abstract

Human visual sensitivity to spatial details declines towards the periphery. Novel image synthesis techniques, so-called foveated rendering, exploit this observation and reduce the spatial resolution of synthesized images for the periphery, avoiding the synthesis of high-spatial-frequency details that are costly to generate but not perceived by a viewer. However, contemporary techniques do not make a clear distinction between the range of spatial frequencies that must be reproduced and those that can be omitted. For a given eccentricity, there is a range of frequencies that are detectable but not resolvable. While the accurate reproduction of these frequencies is not required, an observer can detect their absence if completely omitted. We use this observation to improve the performance of existing foveated rendering techniques. We demonstrate that this specific range of frequencies can be efficiently replaced with procedural noise whose parameters are carefully tuned to image content and human perception. Consequently, these frequencies do not have to be synthesized during rendering, allowing more aggressive foveation, and they can be replaced by noise generated in a less expensive post-processing step, leading to improved performance of the rendering system. Our main contribution is a perceptually-inspired technique for deriving the parameters of the noise required for the enhancement and its calibration. The method operates on rendering output and runs at rates exceeding 200 FPS at 4K resolution, making it suitable for integration with real-time foveated rendering systems for VR and AR devices. We validate our results and compare them to the existing contrast enhancement technique in user experiments.

Towards Attention-aware Rendering for Virtual and Augmented Reality

“…Setup. Acuity-driven foveation algorithms conservatively account for the worst-case scenario of the smallest detectable image detail [Tariq et al 2022]. As seen in our results, distribution of contrast in specific images affects the acceptable foveation intensity.…”

“…Foveated graphics techniques exploit eccentricity-dependent aspects of human vision, such as acuity, to minimize the bandwidth of a graphics system by optimizing bit depth [McCarthy et al 2004], color-fidelity , level-of-detail [Luebke and Hallen 2001;Murphy and Duchowski 2001], or by simply reducing the number of vertices or fragments a graphics processing unit has to sample, ray-trace, shade, or transmit to the display; see [Duchowski et al 2004;Koulieris et al 2019] for a review of this area. Foveated rendering is perhaps the most well-known example of this class of algorithms [Deng et al 2022;Friston et al 2019;Geisler and Perry 1998;Guenter et al 2012;Kaplanyan et al 2019;Patney et al 2016;Sun et al 2017;Tariq et al 2022;Tursun et al 2019]. The perceptual models underlying foveated graphics usually exploit spatial aspects of eccentricity-dependent human vision but, to the best of our knowledge, none of them model cognitive or attentional effects of human vision, which we aim to address with our work.…”

“…It must be noted that gains in real applications are influenced by efficiency of a particular renderer. As our contributions are independent of such design choices, more advanced foveation approaches such as noise-based enhancement [Tariq et al 2022] can be considered for additional gains.…”