A robust fusion method for multiview distributed video coding

Abstract

Distributed video coding (DVC) is a coding paradigm which exploits the redundancy of the source (video) at the decoder side, as opposed to predictive coding, where the encoder leverages the redundancy. To exploit the correlation between views, multiview predictive video codecs require the encoder to have the various views available simultaneously. However, in multiview DVC (M-DVC), the decoder can still exploit the redundancy between views, avoiding the need for inter-camera communication. The key element of every DVC decoder is the side information (SI), which can be generated by leveraging intra-view or inter-view redundancy for multiview video data. In this paper, a novel learning-based fusion technique is proposed, which is able to robustly fuse an inter-view SI and an intra-view (temporal) SI. An inter-view SI generation method capable of identifying occluded areas is proposed and is coupled with a robust fusion system able to improve the quality of the fused SI along the decoding process through a learning process using already decoded data. We shall here take the approach to fuse the estimated distributions of the SIs as opposed to a conventional fusion algorithm based on the fusion of pixel values. The proposed solution is able to achieve gains up to 0.9 dB in Bjøntegaard difference when compared with the best-performing (in a RD sense) single SI DVC decoder, chosen as the best of an inter-view and a temporal SI-based decoder one.

Improved Deep Distributed Light Field Coding

“…Mukati et al [12] propose to use a view synthesis-based approach to synthesize light field views at the decoder utilizing only four key views picked from the four corners of the LF in order to reduce the encoding complexity radically. The results show that leveraging high-quality synthesized views provide competitive RD performance compared to the state-of-the-art DMVC approach [13].…”

“…To achieve a higher RD performance, the transform domain WZ coding is adopted to exploit better the spatial correlation in [11]. A DMVC approach is proposed in [13]. It generates multiple SIs utilizing temporal and inter-view redundancies.…”

View Synthesis-based Distributed Light Field Compression

“…Among the schemes, the fusion scheme utilizing the reciprocal of the residual and the reciprocal of vector magnitude as weights have superior performance compared to the former two fusion schemes. Salmistraro et al [7] propose a DMVC approach which exploits temporal and inter-view redundancies at the decoder side by generating multiple SIs. Moreover, a robust fusion method is employed by fusing likelihoods estimated from each SI.…”

“…In FNV case, as illustrated in Fig. 3 (b), the views are split in a checkerboard pattern allowing to utilize horizontal and vertical adjacent neighbors to estimate SI for WZV in a similar manner as in DMVC [7]. Contrary to DMVC, an additional angular dimension and LF image signal are considered.…”

“…In this work, we build on top of the latest state-of-the-art method in DMVC [7], and we propose improving the estimation at the decoder side. More precisely, we replace the typically employed optical flow [8] or overlapped block mo-tion compensation [9] to generate side information (SI) with a learning-based view synthesis approach, which estimates the scene geometry and inpaints occlusion, to obtain higherquality estimates.…”