View synthesis techniques for 3D video

Tian, Dong; Lai, PoLin; Lopez, Patrick; Gomila, Cristina

doi:10.1117/12.829372

Cited by 168 publications

(153 citation statements)

References 0 publications

Supporting

Mentioning

151

Contrasting

Unclassified

Order By: Relevance

“…To enable synthesis of novel images at arbitrarily chosen viewpoints at decoder, texture-plus-depth format [3] dictates the transmission of texture and depth maps capturing the same 3D scene by one or more closely spaced cameras. The selected viewpoint is synthesized at the receiver via a procedure known as DIBR [4]. DIBR is a pixel-to-pixel mapping such that the reference image pixels are first projected back to the world coordinates using depth map and then reprojected to the virtual image coordinate.…”

Section: System Overviewmentioning

confidence: 99%

“…Figure 1 shows examples of both disocclusion and rounding holes. Rounding holes tend to be small and can be filled easily using conventional filtering techniques [4]. The focus of our paper is on the filling of disocclusion holes.…”

Section: System Overviewmentioning

confidence: 99%

“…Beside conventional color (e.g., RGB) images, depth images (per-pixel distance between physical objects in the 3D scene and a capturing camera) from the same camera viewpoints can also be acquired using active depth sensors like time-of-flight cameras [2]. Transmitting both texture and depth maps from multiple viewpoints-a format known as texture-plus-depth or video-plus-depth [3]-enables a user to freely select virtual viewpoints from which to synthesize novel images for observation via depth-image-based rendering (DIBR) [4]. It has been shown [5] that allowing a user to interactively select different viewpoints from which to observe a 3D scene can greatly enhance the user's depth perception of the scene, improving his overall visual experience.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Joint texture-depth pixel inpainting of disocclusion holes in virtual view synthesis

Reel

Cheung

Wong

et al. 2013

2013 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference

View full text Add to dashboard Cite

Abstract-Transmitting texture and depth maps from one or more reference views enables a user to freely choose virtual viewpoints from which to synthesize images for observation via depth-image-based rendering (DIBR). In each DIBR-synthesized image, however, there remain disocclusion holes with missing pixels corresponding to spatial regions occluded from view in the reference images. To complete these holes, unlike previous schemes that rely heavily (and unrealistically) on the availability of a high-quality depth map in the virtual view for inpainting of the corresponding texture map, in this paper a new Joint TextureDepth Inpainting (JTDI) algorithm is proposed that simultaneously fill in missing texture and depth pixels. Specifically, we first use available partial depth information to compute priority terms to identify the next target pixel patch in a disocclusion hole for inpainting. Then, after identifying the best-matched texture patch in the known pixel region via template matching for texture inpainting, the variance of the corresponding depth patch is copied to the target depth patch for depth inpainting. Experimental results show that JTDI outperforms two previous inpainting schemes that either does not use available depth information during inpainting, or depends on the availability of a good depth map at the virtual view for good inpainting performance.

show abstract

Section: System Overviewmentioning

confidence: 99%

Section: System Overviewmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Joint texture-depth pixel inpainting of disocclusion holes in virtual view synthesis

Reel

Cheung

Wong

et al. 2013

2013 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference

View full text Add to dashboard Cite

show abstract

“…The process of view synthesis is composed of three steps [5], a warping step, in which the samples to the virtual position are warped from input views based on the geometry of the scene; a blending step, in which the warped pictures from each input viewpoint are combined into a single picture; and a hole filling step, in which any remaining holes in the blended picture are filled. The blending is only invoked when there are multiple input viewpoints from which the synthesized view is generated.…”

Section: Background On View Synthesismentioning

confidence: 99%

A trellis-based approach for robust view synthesis

Tian

Vetro

Brand

2011

2011 18th IEEE International Conference on Image Processing

Self Cite

View full text Add to dashboard Cite

View synthesis is an essential function for a number of 3D video applications including freeviewpoint navigation and view generation for auto-stereoscopic displays. Depth Image Based Rendering (DIBR) techniques are typically applied for this purpose. However, the quality of the rendered views is very sensitive to the quality of the depth image. In this paper, a novel trellisbased view synthesis framework is proposed to overcome the above limitations in depth images and reduce artifacts in the rendered picture. Our results demonstrate that the proposed approach offers visible improvements in rendering quality compared to existing view synthesis techniques. IEEE International Conference on Image Processing (ICIP)This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Research Laboratories, Inc.; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Research Laboratories, Inc. All rights reserved. ABSTRACTView synthesis is an essential function for a number of 3D video applications including free-viewpoint navigation and view generation for auto-stereoscopic displays. Depth Image Based Rendering (DIBR) techniques are typically applied for this purpose. However, the quality of the rendered views is very sensitive to the quality of the depth image. In this paper, a novel trellis-based view synthesis framework is proposed to overcome the above limitations in depth images and reduce artifacts in the rendered picture. Our results demonstrate that the proposed approach offers visible improvements in rendering quality compared to existing view synthesis techniques.

show abstract

“…By transmitting both texture maps (color images) and depth maps (per-pixel distance between objects in the 3D scene and the capturing camera) captured from one or more camera view(s), free viewpoint video [1] enables a user the ability to synthesize novel virtual view images via depth-imagebased rendering (DIBR) [2]. In a nutshell, DIBR copies color pixels in the camera-captured view(s) to corresponding pixel locations in the virtual view image, given 3D geometric information provided by the depth map(s).…”

Section: Introductionmentioning

confidence: 99%

Disocclusion hole-filling in DIBR-synthesized images using multi-scale template matching

Reel

Wong

Cheung

et al. 2014

2014 IEEE Visual Communications and Image Processing Conference

View full text Add to dashboard Cite

Abstract-Transmitting texture and depth images of captured camera view(s) of a 3D scene enables a receiver to synthesize novel virtual viewpoint images via Depth-Image-Based Rendering (DIBR). However, a DIBR-synthesized image often contains disocclusion holes, which are spatial regions in the virtual view image that were occluded by foreground objects in the captured camera view(s). In this paper, we propose to complete these disocclusion holes by exploiting the self-similarity characteristic of natural images via nonlocal template-matching (TM). Specifically, we first define self-similarity as nonlocal recurrences of pixel patches within the same image across different scales-one characterization of self-similarity in a given image is the scale range in which these patch recurrences take place. Then, at encoder we segment an image into multiple depth layers using available per-pixel depth values, and characterize self-similarity in each layer with a scale range; scale ranges for all layers are transmitted as side information to the decoder. At decoder, disocclusion holes are completed via TM on a per-layer basis by searching for similar patches within the designated scale range. Experimental results show that our method improves the quality of rendered images over previous disocclusion hole-filling algorithms by up to 3.9dB in PSNR.Index Terms-Free viewpoint video, depth-image-based rendering, image inpainting

show abstract

View synthesis techniques for 3D video

Cited by 168 publications

References 0 publications

Joint texture-depth pixel inpainting of disocclusion holes in virtual view synthesis

Joint texture-depth pixel inpainting of disocclusion holes in virtual view synthesis

A trellis-based approach for robust view synthesis

Disocclusion hole-filling in DIBR-synthesized images using multi-scale template matching

Contact Info

Product

Resources

About