Client-Driven Selective Streaming of Multiview Video for Interactive 3DTV

Ortega

IEEE Trans. on Image Process.

2011

129

Abstract-While much of multiview video coding focuses on the rate-distortion performance of compressing all frames of all views for storage or non-interactive video delivery over networks, we address the problem of designing a frame structure to enable interactive multiview streaming, where clients can interactively switch views during video playback. Thus, as a client is playing back successive frames (in time) for a given view, it can send a request to the server to switch to a different view while continuing uninterrupted temporal playback. Noting that standard tools for random access (i.e., I-frame insertion) can be bandwidth-inefficient for this application, we propose a redundant representation of I-, P-, and "merge" frames, where each original picture can be encoded into multiple versions, appropriately trading off expected transmission rate with storage, to facilitate view switching. We first present ad hoc frame structures with good performance when the view-switching probabilities are either very large or very small. We then present optimization algorithms that generate more general frame structures with better overall performance for the general case. We show in our experiments that we can generate redundant frame structures offering a range of tradeoff points between transmission and storage, e.g., outperforming simple I-frame insertion structures by up to 45% in terms of bandwidth efficiency at twice the storage cost.Index Terms-Media interaction, multiview video coding, video streaming.

Section: Interactive Multiview Video Streamingmentioning

confidence: 99%

Interactive Streaming of Stored Multiview Video Using Redundant Frame Structures

Ortega

IEEE Trans. on Image Process.

2011

129

“…[2], coding schemes that facilitate quick viewpoint switching [23], [24], [25], and that combine multiple texture/color and depth map streams to achieve free viewpoint streaming, including of virtual viewpoints [26], [5], [27], [28], [29]. In the context of 3D-TV, both multi-layered solutions [30] and prefetching information of alternative viewpoints [31] have been shown useful.…”

Section: H Bandwidth Variationsmentioning

confidence: 99%

Optimized Adaptive Streaming of Multi-video Stream Bundles

Carlsson

Eager

Krishnamoorthi

et al. 2017

IEEE Trans. Multimedia

Abstract-In contrast to traditional video, multi-view video streaming allows viewers to interactively switch among multiple perspectives provided by different cameras. One approach to achieving such a service is to encode the video from all of the cameras into a single stream, but this has the disadvantage that only a portion of the received video data will be used, namely that required for the selected view at each point in time. In this paper we introduce the concept of a "multi-video stream bundle" that consists of multiple parallel video streams that are synchronized in time, each providing the video from a different camera capturing the same event or movie. For delivery we leverage the adaptive features and time-based chunking of HTTPbased Adaptive Streaming (HAS), but now employing adaptation in both content and rate. Users are able to change their viewpoint on-demand and the client player adapts the rate at which data is retrieved from each stream based on the user's current view, the probabilities of switching to other views, and the user's current bandwidth conditions. A crucial component of such a system is the prefetching policy. For this we present an optimization model as well as a simpler heuristic that can balance the playback quality and the probability of playback interruptions. After analytically and numerically characterizing the optimal solution, we present a prototype implementation and sample results. Our prefetching and buffer management solution is shown to provide close to seamless playback switching when there is sufficient bandwidth to prefetch the parallel streams.

“…For interactive streaming of stored multiview videos, the two-layer approach proposed in [3], [17] can be one solution, where coarse and fine quality layers of several views are grouped and pre-encoded. During actual streaming, a subset of views of low quality plus two views of high quality, carefully selected based on user's behavioral prediction, would then be sent to the client.…”

Section: B Interactive Multiview Video Streamingmentioning

confidence: 99%

“…Several prototypes of such multiview video systems [1], [3] have demonstrated an improved viewing experience via this view-switching media interaction.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Delay-Cognizant Interactive Streaming of Multiview Video With Free Viewpoint Synthesis

Xiu

Liang

2012

IEEE Trans. Multimedia

In interactive multiview video streaming (IMVS), a client receives and observes one of many available viewpoints of the same scene, and periodically requests from server view-switches to neighboring views, as the video is played back in time uninterruptedly. One key technical challenge is to design a frame coding structure that facilitates periodic view-switching, and achieves an optimal tradeoff between storage cost and expected transmission rate. In this paper, we first propose three significant improvements over existing IMVS system, and then study the corresponding frame structure optimization. First, using depth-image-based rendering, the new IMVS system enables free viewpoint switching, i.e., by encoding and transmitting both texture and depth maps of captured views, a client can select and synthesize any virtual view from an almost continuum of viewpoints between the left-most and right-most captured views. Second, IMVS system adopts a more realistic Markovian view-switching model with memory that more accurately captures user behaviors than previous memoryless models. View-switching model is used in predicting client's future view-switching patterns. Third, assuming that the round-trip-time (RTT) delay during server-client communication is non-negligible, during an IMVS session, IMVS system additionally transmits redundant frames RTT into future playback, so that zero-delay view-switching can be achieved. Given these improvements, we formalize a new join optimization of the frame coding structure, transmission schedule, and quantization parameters of the texture and depth maps of multiple camera views. We propose an iterative algorithm to achieve fast and near-optimal solutions.The convergence of the algorithm is also demonstrated. Experimental results show that the proposed optimized rate allocation method requires 38% less transmission rate than the fixed rate allocation scheme. In addition, with the same storage, the transmission rate of the optimized frame structure can be up to 55% lower than that of I-frame-only structure, and 27% lower than that of the structure without distributed source coding (DSC) frames. Index Termsmultiview video, video streaming, media interaction, view synthesis