A Hole Filling Approach Based on Background Reconstruction for View Synthesis in 3D Video

Depth-Image-Based Rendering (DIBR) is a fundamental technology in several 3D-related applications, such as Free viewpoint video (FVV), Virtual Reality (VR) and Augmented Reality (AR). However, new challenges have also been brought in assessing the quality of DIBR-synthesized views since this process induces some new types of distortions, which are inherently different from the distortion caused by video coding. In this paper, we present a new DIBR-synthesized image database with the associated subjective scores. We also test the performances of the state-of-the-art objective quality metrics on this database. This work focuses on the distortions only induced by different DIBR synthesis methods. Seven state-of-the-art DIBR algorithms, including interview synthesis and single view based synthesis methods, are considered in this database. The quality of synthesized views was assessed subjectively by 41 observers and objectively using 14 state-of-the-art objective metrics. Subjective test results show that the interview synthesis methods, having more input information, significantly outperform the single view based ones. Correlation results between the tested objective metrics and the subjective scores on this database reveal that further studies are still needed for a better objective quality metric dedicated to the DIBR-synthesized views.

show abstract

“…Luo et al proposed a hole filling approach for DIBR systems based on background reconstruction [44]. As shown in Fig.…”

Section: Luo's Methodsmentioning

confidence: 99%

A Benchmark of DIBR Synthesized View Quality Assessment Metrics on a New Database for Immersive Media Applications

Tian

Zhang

Morin

et al. 2019

IEEE Trans. Multimedia

View full text Add to dashboard Cite

show abstract

“…The foreground objects are detected and removed, then motion compensation and modified GMMs are applied to construct a stable background. Results indicated that a clean background without artifacts of foreground objects can be generated by using the proposed background model, so that the blurry effect or artifacts in disoccluded regions can be eliminated and the sharp edges along the foreground boundaries can be preserved with realistic appearance [24].…”

Section: Related Workmentioning

confidence: 97%

Reliability-Based View Synthesis for Free Viewpoint Video

Deng

Wang

2018

Applied Sciences

View full text Add to dashboard Cite

Featured Application: View synthesis technology is important for free viewpoint video (FVV) and multiview video coding (MVC). It is a practical approach to reduce storage and transmission bandwidth for multiview videos.Abstract: View synthesis is a crucial technique for free viewpoint video and multi-view video coding because of its capability to render an unlimited number of virtual viewpoints from adjacent captured texture images and corresponding depth maps. The accuracy of depth maps is very important to the rendering quality, since depth image-based rendering (DIBR) is the most widely used technology among synthesis algorithms. There are some issues due to the fact that stereo depth estimation is error-prone. In addition, filling occlusions is another challenge in producing desirable synthesized images. In this paper, we propose a reliability-based view synthesis framework. A depth refinement method is used to check the reliability of depth values and refine some of the unreliable pixels, and an adaptive background modeling algorithm is utilized to construct a background image aiming to fill the remaining empty regions after a proposed weighted blending process. Finally, the proposed approach is implemented and tested on test video sequences, and experimental results indicate objective and subjective improvements compared to previous view synthesis methods.

show abstract

“…Li Yao et al [6] combine depthbased image fusion with direct image fusion to decrease the ghost effect. Luo et al [7] extracted and separated the foreground objects and filled the holes with a relatively stable background obtained by the Gaussian mixture model. Although this method can effectively avoid artifacts, the construction of the background is very complicated and the parameters of the model are difficult to choose.…”

Section: Related Workmentioning

confidence: 99%

View synthesis based on spatio-temporal continuity

Yao

2019

J Image Video Proc.

View full text Add to dashboard Cite

Free viewpoint video is generated on the basis of a video plus depth (V+D) virtual point rendering framework. Because of the limited bandwidth of video transmission, depth-image-based rendering (DIBR) has become a common method. Most DIBR methods not only have holes and ghost artifacts but also have problems with time continuity, leading to frequent flickers. In this paper, we make full use of time domain information in video sequences and adjacent frames to extract the static background image of the whole scene. Furthermore, we propose a weighted-fusion hole-filling method based on static background to fill holes and maintain time continuity. Experimental results show that the proposed method can improve the quality of virtual view images and strengthen spatio-temporal continuity.

show abstract

A Hole Filling Approach Based on Background Reconstruction for View Synthesis in 3D Video

Cited by 52 publications

References 27 publications

A Benchmark of DIBR Synthesized View Quality Assessment Metrics on a New Database for Immersive Media Applications

A Benchmark of DIBR Synthesized View Quality Assessment Metrics on a New Database for Immersive Media Applications

Reliability-Based View Synthesis for Free Viewpoint Video

View synthesis based on spatio-temporal continuity

Contact Info

Product

Resources

About