Bypassing Depth Maps Transmission For Immersive Video Coding

Abstract

This paper addresses several downsides of the system under development in MPEG-I for coding and transmission of immersive media. We present a solution, which enables Depth-Image-Based Rendering for immersive video applications, while lifting the requirement of transmitting depth information. Instead, we estimate the depth information on the client-side from the transmitted views. The approach leads to an impressive rate saving (37.3% in average). Preserving perceptual quality in terms of MS-SSIM of synthesized views, it yields to 24.6% rate reduction for the same quality of reconstructed views after residue transmission under the MPEG-I common test conditions. Simultaneously, the required pixel rate, i.e. the number of pixels processed per second by the decoder, is reduced by 50% for any test sequence. To the author's knowledge, this is the first time that such an approach is under consideration in the context of immersive video coding.

Immersive Video Coding: Should Geometry Information Be Transmitted as Depth Maps?

“…The depth maps are not transmitted, the depth estimator is moved to the decoder side and depth maps D + are estimated from decoded textures T * to synthesize S DSDE . A first analysis of this system compared to ESDE is presented in [7] for full views and in [16] for patch atlases. In contrast to ESDE, little investigation [17] and improvement have been done towards the DSDE system up to this date.…”

“…We have shown in our previous work, that most of the downsides of ESDE are solved by completely bypassing the transmission of depth maps and then performing the depth estimation at the decoder side using the decompressed texture views. This approach is called decoder side depth estimation (DSDE) [7]. In particular, this approach enables a significant improvement in coding efficiency because most of the geometry can be estimated from the decoded textures at the decoder.…”

“…Furthermore, DSDE reduces the pixel rate as depth maps do not need to be decoded. However, this method requires an additional depth estimation stage in the decoder and may produce depth map inaccuracies [7]. This raises the following question: what part of the geometry should be sent (at the price of a rate cost) and what should be estimated at the decoder (at the price of a possibly degraded view synthesis quality and a higher complexity)?…”

Depth Map Refinement for Immersive Video

“…Moreover, because the proposal does not utilize any color information, its usability is very high for decoder-side depth estimation [46]. The quality of depth maps estimated using encoded input views can vary significantly, depending on the available bitrate [47].…”

Patch Decoder-Side Depth Estimation In Mpeg Immersive Video

“…This approach is based on reducing the depth map transmission from the encoder side, and on moving a part of the depth estimation process to the decoder side. Recently, it has been shown that decoderside depth estimation (DSDE) can efficiently recover the depth maps at the decoder side while reducing the pixel rate by 50% and saving 37.3% in terms of Bjøntegaard delta (BD) rate metric [10], when applied to the aforementioned MV-HEVC + VVS framework [11], [2]. Hence, the DSDE approach was studied in the case of full views, where the depth estimation process took advantage of many views that are available at the decoder.…”