Delay-Cognizant Interactive Streaming of Multiview Video With Free Viewpoint Synthesis

Abstract

In interactive multiview video streaming (IMVS), a client receives and observes one of many available viewpoints of the same scene, and periodically requests from server view-switches to neighboring views, as the video is played back in time uninterruptedly. One key technical challenge is to design a frame coding structure that facilitates periodic view-switching, and achieves an optimal tradeoff between storage cost and expected transmission rate. In this paper, we first propose three significant improvements over existing IMVS system, and then study the corresponding frame structure optimization. First, using depth-image-based rendering, the new IMVS system enables free viewpoint switching, i.e., by encoding and transmitting both texture and depth maps of captured views, a client can select and synthesize any virtual view from an almost continuum of viewpoints between the left-most and right-most captured views. Second, IMVS system adopts a more realistic Markovian view-switching model with memory that more accurately captures user behaviors than previous memoryless models. View-switching model is used in predicting client's future view-switching patterns. Third, assuming that the round-trip-time (RTT) delay during server-client communication is non-negligible, during an IMVS session, IMVS system additionally transmits redundant frames RTT into future playback, so that zero-delay view-switching can be achieved. Given these improvements, we formalize a new join optimization of the frame coding structure, transmission schedule, and quantization parameters of the texture and depth maps of multiple camera views. We propose an iterative algorithm to achieve fast and near-optimal solutions.The convergence of the algorithm is also demonstrated. Experimental results show that the proposed optimized rate allocation method requires 38% less transmission rate than the fixed rate allocation scheme. In addition, with the same storage, the transmission rate of the optimized frame structure can be up to 55% lower than that of I-frame-only structure, and 27% lower than that of the structure without distributed source coding (DSC) frames. Index Termsmultiview video, video streaming, media interaction, view synthesis

Hypervideos and Interactive Multimedia Presentations

“…Navigation in the presentations can be performed using VCR actions or jumps on a timeline. Basic multimedia presentations can be created with SMIL Builder [Bouyakoub and Belkhir 2011], GRiNS [Bulterman et al 1998], SMILAuthor [Yang and Yang 2003]/SMILAuthor2 [Yang et al 2008], the Synchronization Editor [Blakowski et al 1992], TYRO [MacNeil 1991], MPRES Author [Wong et al 1997], Java-Assisted SMIL (JAS) [Dai et al 2006 [Kelly et al 1995;Katkere et al 1996;Maugey and Frossard 2011;Miller et al 2011;Xiu et al 2012], tools for video annotation using video analysis, m-ontomatannotizer [Petridis et al 2006], or iVAT [Bianco et al 2013], and social video/social TV [Wang et al 2012;Shin 2013] are not taken into account in this survey. They provide certain interactive or nonlinear features, but they are not closely related to the types of extended videos described in this section.…”

Optimal layered representation for adaptive interactive multiview video streaming

“…In general, the limited bandwidth problem in IMVS has been approached by proposing some coding/prediction structure optimization mechanisms for the compression of multiview video data. In [3], [4] and [5], the authors have studied the prediction structures based on redundant P-and DSC-frames (distributed source coding) that facilitate a continuous view-switching by trading off the transmission rate and the storage capacity. To save transmission bandwidth, different interview prediction structures are proposed in [6] to code in different ways each multiview video dataset, in order to satisfy different rate-distortion requirements.…”

“…. Based on the above observations, (5) and (6), this minimum distortion can be defined in a recursive way as follows:…”