Multiple Description Coding and Recovery of Free Viewpoint Video for Wireless Multi-Path Streaming

Abstract

By transmitting texture and depth videos captured from two nearby camera viewpoints, a client can synthesize via depth-image-based rendering (DIBR) any freely chosen intermediate virtual view of the 3D scene, enhancing the user's perception of depth. During wireless network transmission, burst packet losses can corrupt the transmitted texture and depth videos and degrade the synthesized view quality at the client. In this paper, we propose a multiple description coding system for multi-path transmission of free-viewpoint video, with joint interview and temporal description recovery capability. In particular, we encode separately the even frames of the left view and the odd frames of the right view, and transmit them as one description on one path. The second description comprises the remaining frames in the two views and is transmitted over a second path. If the receiver receives only one description due to burst loss in the other path, the missing frames in the other description are partially reconstructed using our frame recovery procedure. First, we construct two recovery candidates for each lost pixel in a frame. The first candidate is generated via temporal super-resolution from its predecessor and successor frames in the same view. The second candidate is generated via DIBR from the received frame of the same time instance in the other view. Next, we select the best pixel candidates one patch at a time, where an image patch corresponds to a neighborhood of pixels with similar depth values in the 3D scene. Near-optimal source and channel coding rates for each description are selected using a branch-and-bound method, for given transmission bandwidth on each path. Experimental results show that our system can outperform a traditional single-description/single-path transmission scheme by up to 5.5 dB in Peak Signal-to-Noise Ratio (PSNR) of the synthesized intermediate view at the client.Index Terms-Free-viewpoints video, multiple description coding, wireless multi-path streaming, depth-image-based rendering.

Adaptive Multiview Video Delivery Using Hybrid Networking

“…The aforementioned works are effective in the Content Delivery Network (CDN) type applications, where a number of local content servers are maintained to support a larger base of users. Another recent work addressed the interactive delivery of the stereoscopic scene of interest (out of a multi-view video) over wireless transmission networks in an error-resilient way by exploiting Multiple Description Coding (MDC) [24]. The proposed approach generates a couple of descriptions from the textureplus-depth videos of the left and the right views of the associated views of interest, which are then transmitted through separate wireless channels to exploit path diversity.…”

Reliable 3D video streaming considering region of interest

“…One major problem with the stream of a video is errors that happen due to the unreliable communication in both wired or wireless networks [7][8][9][10][11][12][13][14]. In the wired networks, errors can occur due to packet loss, corruption, congestion, and large packet delay, whereas in the wireless networks, unreliable communication can stem from temperature noise and interference that exist in the physical environment.…”

“…With the temporal MDC method, usually, two descriptions are produced in order to avoid a drop in the coding efficiency. The drop in the coding efficiency is reflected when more than two descriptions are used because the distance between the assigned frames to each description is increasing resulting in the motion prediction being less effective [8,16]. When the network is very noisy, a higher number of descriptions are required.…”

“…Consequently, one can state that the neglected pixels during downsampling can be better predicted. Liu et al utilized the fact of having similar depth values of pixels for real objects and introduced a texture block partitioning algorithm in order to perform their MDC algorithm for wireless multi-path streaming [8].…”

Cross-Layer Management for Multiple Adaptive Streaming Clients in Wireless Home Networks

“…In our study, the difference is that multiple TCP connections continuously compete with one another. In [6], [7], free viewpoint video is delivered by multi-path streaming. Especially, a video is represented by multiple description coding (MDC) to cope with packet losses.…”