Real-Time Depth Video-Based Rendering for 6-DoF HMD Navigation and Light Field Displays

Abstract

This paper presents a novel approach to provide immersive free navigation with 6 Degrees of Freedom in real-time for natural and virtual scenery, for both static and dynamic content. Stemming from the state-of-the-art in Depth Image-Based Rendering and the OpenGL pipeline, this new View Synthesis method achieves free navigation at up to 90 FPS and can take any number of input views with their corresponding depth maps as priors. Video content can be played thanks to GPU decompression, supporting free navigation with full parallax in real-time. To render a novel viewpoint, each selected input view is warped using the camera pose and associated depth map, using an implicit 3D representation. The warped views are then blended all together to generate the chosen virtual view. Various view blending approaches specifically designed to avoid visual artifacts are compared. Using as few as four input views appears to be an optimal trade-off between computation time and quality, allowing to synthesize high-quality stereoscopic views in real-time, offering a genuine immersive virtual reality experience. Additionally, the proposed approach provides high-quality rendering of a 3D scenery on holographic light field displays. Our results are comparable -objectively and subjectively -to the state of the art view synthesis tools NeRF and LLFF, while maintaining an overall lower complexity and real-time rendering.

View Synthesis Tool for VR Immersive Video

“…In other words, the input pixels should be mapped to more than one pixel to compensate for this effect. This can be done using superpixels [20], adapting the pixels size to the camera movement [21], or linking neighboring pixels for rasterization [15,16] (chosen solution in RVS: adjacent pixels are grouped into triangles that are colorized).…”

“…A depth map refinement [26,27] is one way to solve this problem. Another is to choose weighted blending coefficients based on the reliability of each input image [11,16] (chosen solution in RVS).…”

“…When the input images are warped to the target viewpoint, the results need to be blended into one single image. For a given pixel, the final output color c is the weighted mean of the color c i of each warped input: where w i is a weight representing the quality of a triangle [16], prioritizing foreground objects and highest quality triangles.…”

“…Displaying a dynamic scene in VR requires real-time view synthesis, preferably at 90 frames per second and at a minimum of 30 frames per second for each eye. However, the processing time depends on the number of input images and their resolutionsince their pixels form the meshresulting in different frame rates [16] (see Table 1). Using a NVIDIA GTX 1080TI GPU, around four input images at a full HD resolution can be processed to obtain a high visual quality while reaching realtime navigation.…”

“…This chapter provides comprehensive elements to bring photographs of a natural scene to the third dimension, for example, making the captured scene immersive, through holography or virtual reality (VR). The presented 3D rendering technique differs from traditional computer graphics by its input-instead of modeling 3D objects with their geometry and materials that interact with light sources, we use photographs that are warped to follow the viewer gaze direction using the reference view synthesis (RVS) [14][15][16] software that follows the view synthesis process of Figure 1.…”