Polynomial Image-Based Rendering for non-Lambertian Objects

Abstract

Non-Lambertian objects present an aspect which depends on the viewer's position towards the surrounding scene. Contrary to diffuse objects, their features move non-linearly with the camera, preventing rendering them with existing Depth Image-Based Rendering (DIBR) approaches, or to triangulate their surface with Structure-from-Motion (SfM). In this paper, we propose an extension of the DIBR paradigm to describe these non-linearities, by replacing the depth maps by more complete multi-channel "non-Lambertian maps", without attempting a 3D reconstruction of the scene. We provide a study of the importance of each coefficient of the proposed map, measuring the tradeoff between visual quality and data volume to optimally render non-Lambertian objects. We compare our method to other stateof-the-art image based rendering methods and outperform them with promising subjective and objective results on a challenging dataset.

Depth Image-Based Rendering of Non-Lambertian Content in MPEG Immersive Video

“…RVS 4.0 is based on the observation that features observed on the surface of non-Lambertian objects move non-linearly with respect to a linear camera movement, contrary to diffuse objects, which parallax is inversely proportional to their distance to the camera. As described in [24], two polynomials of degree up to three model this displacement.…”

“…The polynomials are computed pixel-wise given the optical flow between input images (exactly like a depth map computation). We provide an ablation study on the required precision (number of bits) of the polynomials coefficients and their degree in [24].…”

“…A call for new test material including non-Lambertian objects [18] has brought to MPEG-I new challenging video [19,20,21,22] that classical DIBR algorithms fail to render with high fidelity. In order to handle such scenes with a DIBRbased method, RVS has been extended to RVS 4.0 that replaces traditional disparity maps by non-Lambertian polynomial maps, which describe the non-linear displacement of features visible on non-Lambertian surfaces [23,24]. To test the performance of the view synthesizer on this specific content, an exploration experiment has been designed to compare our results with single depth map DIBR [25,26].…”

View Synthesis Tool for VR Immersive Video

“…When such objects, socalled non-Lambertian, are present in the scene, the linear hypothesis in pixel displacement in the function of the camera displacement is not valid anymore. Adapting the DIBR principles to non-Lambertian objects is nevertheless possible by exploiting additional information, such as structure, normal, and indexes of refraction [28], or a more accurate approximation of the pixel displacement [29][30][31] (chosen solution in RVS).…”

“…Alternatively, to model the non-Lambertian surface in itself, it is possible to track its feature movements on the surface [29,33,34]. DIBR can be generalized to non-Lambertian objects by replacing the usual depth maps with the coefficients of a polynomial approximating the non-Lambertian features displacement [30,31]. To clearly understand what this means, let us start with what happens for diffuse objects, where for a lateral camera movement x, y ðÞ , the new position u, v ðÞ of a pixel u 0 , v 0 ðÞ is given by:…”