On modeling the rendering error in 3D video

Abstract

We propose an analytical model to estimate the rendering quality in 3D video. The model relates errors in the depth images to the rendering quality, taking into account texture image characteristics, texture image quality, the camera configuration and the rendering process. Specifically, we derive position (disparity) errors from the depth errors, and the probability distribution of the position errors is used to calculate the power spectral density of the rendering errors. Experiment results with video sequences and coding/ rendering tools used in MPEG 3DV activities show that the model can accurately estimate the synthesis noise up to a constant offset. Thus, the model can be used to estimate the change in rendering quality for different system designs. IEEE International Conference on Image Processing (ICIP)This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Research Laboratories, Inc.; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Research Laboratories, Inc. All rights reserved. ON MODELING THE RENDERING ERROR IN 3D VIDEO Ngai-Man Cheung, Dong Tian, Anthony Vetro, Huifang SunMitsubishi Electric Research Laboratories ABSTRACT We propose an analytical model to estimate the rendering quality in 3D video. The model relates errors in the depth images to the rendering quality, taking into account texture image characteristics, texture image quality, the camera configuration and the rendering process. Specifically, we derive position (disparity) errors from the depth errors, and the probability distribution of the position errors is used to calculate the power spectral density of the rendering errors. Experiment results with video sequences and coding/rendering tools used in MPEG 3DV activities show that the model can accurately estimate the synthesis noise up to a constant offset. Thus, the model can be used to estimate the change in rendering quality for different system designs.

Novel distortion metric for depth coding of 3D video

“…Takahashi [10] proposed an optimized view interpolation scheme based on frequency domain analysis of depth map error. Cheung et al [11] proposed to estimate the synthesis quality using power spectral density (PSD).…”

Synthesis distortion estimation in 3D video using frequency and spatial analysis

“…This frequency domain analysis assumed that the underlying image signals are spatial invariant (i.e., wide-sense stationary), which we found that in the current application this would cause rather significant estimation discrepancy (In [16], we used a sequence specific constant to compensate this discrepancy). Specifically, across strong texture edges the video contents change much more quickly than the non-edge regions, which does not agree with the spatial invariant assumption.…”

“…Let V = U − W be the noise in the rendering output due to coding errors in texture/depth images. In [16], we show that under reasonable assumptions the total synthesis noise power (E[V 2 ]) can be estimated by summing two components: one is the synthesis noise power due to texture image coding (E[N 2 ]), the other is the synthesis noise power due to depth image coding…”

“…First, in frame warping, pixels are copied from X l to form an intermediate frame U l , from position (m ′ , n) to (m, n). We assume the cameras are rectified and arranged linearly, and there exists only horizontal disparity given by m − m ′ determined by the depth images, camera parameters and camera distance [16]. Likewise, pixels are copied from Xr to form the intermediate frame Ur.…”

“…In our previous work [16], we proposed to estimate the synthesis quality using power spectral density (PSD). This assumed that the underlying video frame signals are spatial invariant.…”