View synthesis prediction for multiview video coding

Yea, Sehoon; Vetro, Anthony

doi:10.1016/j.image.2008.10.007

Cited by 74 publications

(32 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Color and depth images from multiple camera viewpoints are encoded together, and at the decoder one can synthesize novel intermediate viewpoint images via DIBR [5]. In this kind of representation, an alternative to translational disparity prediction is view-synthesis-based prediction [24]- [26], where a synthesized version of a target view is used for predictive coding. Our proposal is an example of this class of view-synthesis-based methods for inter-view predictive coding, but combines inpainting and advanced transform coding in a "hard-to-easy" block order for improved compression performance.…”

Section: A Multiview Image and Video Codingmentioning

confidence: 99%

Encoder-Driven Inpainting Strategy in Multiview Video Compression

Gao

Cheung

Maugey

et al. 2016

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

Abstract-In free viewpoint video systems, a user has the freedom to select a virtual view from which an image of the 3D scene is rendered, and the scene is commonly represented by color and depth images of multiple nearby viewpoints. In such representation, there exists data redundancy across multiple dimensions: 1) a 3D voxel may be represented by pixels in multiple viewpoint images (inter-view redundancy); 2) a pixel patch may recur in a distant spatial region of the same image due to self-similarity (inter-patch redundancy); and 3) pixels in a local spatial region tend to be similar (inter-pixel redundancy). It is important to exploit these redundancies during interview prediction toward effective multiview video compression. In this paper, we propose an encoder-driven inpainting strategy for inter-view predictive coding, where explicit instructions are transmitted minimally, and the decoder is left to independently recover remaining missing data via inpainting, resulting in lower coding overhead. In particular, after pixels in a reference view are projected to a target view via depth-image-based rendering at the decoder, the remaining holes in the target view are filled via an inpainting process in a block-by-block manner. First, blocks are ordered in terms of difficulty-to-inpaint by the decoder. Then, explicit instructions are only sent for the reconstruction of the most difficult blocks. In particular, the missing pixels are explicitly coded via a graph Fourier transform or a sparsification procedure using discrete cosine transform, leading to low coding cost. For blocks that are easy to inpaint, the decoder independently completes missing pixels via template-based inpainting. We apply our proposed scheme to frames in a prediction structure defined by JCT-3V where inter-view prediction is dominant, and experimentally we show that our scheme achieves up to 3-dB gain in peak-signal-to-noise-ratio in reconstructed image quality over a comparable 3D-High Efficiency Video Coding implementation using fixed 16 × 16 block size.

show abstract

Section: A Multiview Image and Video Codingmentioning

confidence: 99%

Encoder-Driven Inpainting Strategy in Multiview Video Compression

Gao

Cheung

Maugey

et al. 2016

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

show abstract

“…In Nokia 3DV-TM the projected image is included in the reference picture list(s) and serves as a reference for motion compensated prediction (MCP). The VSP implemented in Nokia-3DV is similar to that presented in [15]. However, in order to keep the syntax of the macroblock coding layer unchanged, Nokia 3DV-TM does not include specific VSP skip and direct modes as in [15].…”

Section: B View Synthesis Prediction (Vsp)mentioning

confidence: 99%

“…The original design of VSP in Nokia 3DV-TM was implemented with a forward view synthesis approach (F-VSP) [15]. This process is considered to be demanding in terms of memory use and processing power.…”

Section: B Depth-based Enhanced Texture Coding Toolsmentioning

confidence: 99%

Multiview-Video-Plus-Depth Coding Based on the Advanced Video Coding Standard

Hannuksela

Rusanovskyy

et al. 2013

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

“…This is especially the case for the development of systems that consider 100 [3] or 1600 [4] views, for example, for navigation. This can be achieved with interview prediction techniques that rely on disparity vector fields [5], [6] or depth maps [7], [8], [9].…”

Section: Introductionmentioning

confidence: 99%

Reference View Selection in DIBR-Based Multiview Coding

Maugey

Petrazzuoli

Frossard

et al. 2016

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

Abstract-Augmented reality, interactive navigation in 3D scenes, multiview video and other emerging multimedia applications require large sets of images hence larger data volumes and increased resources compared to traditional video services. The significant increase of the number of images in multiview systems leads to new challenging problems in data representation and data transmission to provide high quality of experience on resource-constrained environments. In order to reduce the size of the data, different multi view video compression strategies have been proposed recently. Most of them use the concept of reference or key views that are used to estimate other images when there is high correlation in the dataset. In such coding schemes, the two following questions become fundamental: i) how many reference views have to be chosen for keeping a good reconstruction quality under coding cost constraints? ii) where to place these key views in the multiview dataset? As these questions are largely overlooked in the literature, we study the reference view selection problem and propose an algorithm for the optimal selection of reference views in multiview coding systems. Based on a novel metric that measures the similarity between the views, we formulate an optimization problem for the positioning of the reference views such that both the distortion of the view reconstruction and the coding rate cost are minimized. We solve this new problem with a shortest path algorithm that determines both the optimal number of reference views and their positions in the image set. We experimentally validate our solution in a practical multiview distributed coding system and in the standardized 3D-HEVC multi view coding scheme. We show that considering the 3D scene geometry in the reference view positioning problem brings significant rate-distortion improvements and outperforms traditional coding strategy that simply selects key frames based on the distance between cameras.Index Terms-Multiview distributed coding, key view positioning, inter-view correlation, view synthesis, multiview image coding

show abstract

View synthesis prediction for multiview video coding

Cited by 74 publications

References 15 publications

Encoder-Driven Inpainting Strategy in Multiview Video Compression

Encoder-Driven Inpainting Strategy in Multiview Video Compression

Multiview-Video-Plus-Depth Coding Based on the Advanced Video Coding Standard

Reference View Selection in DIBR-Based Multiview Coding

Contact Info

Product

Resources

About