Immersive light field video with a layered mesh representation

Abstract

We present a system for capturing, reconstructing, compressing, and rendering high quality immersive light field video. We accomplish this by leveraging the recently introduced DeepView view interpolation algorithm, replacing its underlying multi-plane image (MPI) scene representation with a collection of spherical shells that are better suited for representing panoramic light field content. We further process this data to reduce the large number of shell layers to a small, fixed number of RGBA+depth layers without significant loss in visual quality. The resulting RGB, alpha, and depth channels in these layers are then compressed using conventional texture atlasing and video compression techniques. The final compressed representation is lightweight and can be rendered on mobile VR/AR platforms or in a web browser. We demonstrate light field video results using data from the 16-camera rig of [Pozo et al. 2019] as well as a new low-cost hemispherical array made from 46 synchronized action sports cameras. From this data we produce 6 degree of freedom volumetric videos with a wide 70 cm viewing baseline, 10 pixels per degree angular resolution, and a wide field of view, at 30 frames per second video frame rates. Advancing over previous work, we show that our system is able to reproduce challenging content such as view-dependent reflections, semi-transparent surfaces, and near-field objects as close as 34 cm to the surface of the camera rig.

Real-World Virtual Reality With Head-Motion Parallax

“…This requires rendering from a large SSD storage RAID on a high-end GPU. In contrast, we demonstrate two systems that produce a compelling HMP experience rendered on a single GPU using intermediate representations that store just a few texture-plus-depth panoramas and can be generated in less than 2 CPU-hours (compared, e.g., to 28.5 CPU-hours for the format proposed by Broxton et al [7]). Our representations are constructed from 28 input images (compared to over four thousand for the approach in Luo et al [16]).…”

“…Since the cameras are all in one horizontal plane, such rigs do not catch enough information to support vertical head motion. With this in mind, spherical camera rigs have been proposed more recently [6], [7]. However, a seated VR viewer typically has a much larger horizontal than vertical range of head motion [8].…”

“…The approach focused exclusively on extrapolating from narrow baseline RGBD input. Broxton et al [7] demonstrated high-quality par-allax rendering even for challenging scenes from multi-sphere and layered mesh representations. However, the approach requires a spherical lightfield capture, and constructing the intermediate format from the captured images is computationally expensive.…”

Depth Map Refinement for Immersive Video

“…The researchers are still mainly focused on multiview video plus depth (MVD) representation [35], therefore, further considerations presented in this paper also concern MVD. Of course, multi-plane images (MPI) [7] and their variants [22] are gaining much attention, nevertheless, in these representations, depth information is still present, but in another form.…”

Adaptive View Sampling for Efficient Synthesis of 3D View Using Calibrated Array Cameras

“…Furthermore, the diffraction or interferenceof light can degrade the quality of the results of the reconstruction, rendering, or depth estimation. The quality of the light field imaging depends on both the number of images and the density of the light field, so the interpolating view is one of the most important tasks for the light field camera [37]. Interpolation is considered a simple and efficient method to increase the resolution of an image, so it is widely used in image processing.…”