Reliability-Based View Synthesis for Free Viewpoint Video

Deng, Zengming; Wang, Mingjiang

doi:10.3390/app8050823

Cited by 8 publications

(8 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For video sequence rendering, some methods use background modeling to prevent the flicker between adjacent frames. The background model obtained in the current frame can be used to fill the disocclusions in other frames [4], [40], [41]. This is effective for still background scene.…”

Section: G Discussionmentioning

confidence: 99%

Local Foreground Removal Disocclusion Filling Method for View Synthesis

Liang

Chen

et al. 2020

IEEE Access

View full text Add to dashboard Cite

View synthesis is an effective method to generate the contents of multiple views based on a limited number of reference views, which can be used in 2D to 3D conversion, free viewpoint video and multiview video rendering. Depth-image-based rendering (DIBR) is a practical technique to generate virtual view by using a 2D reference view and its depth image. However, a critical problem in DIBR process is that disocclusions might be produced in the synthesized image because the background occluded by the foreground objects in the reference view may be exposed in the virtual view. In this paper, a local foreground removal method is proposed for disocclusion filling. Morphology-based depth image preprocessing is performed before DIBR, aiming to correct the depth value of the ghosts and remove ghost artifacts. In the synthesized virtual image, pixels on the disocclusion edge are identified and classified. Then they are positioned in the reference image by inverse 3D warping. Local foreground regions that occlude the corresponding background are removed from both the reference image and its depth image based on the disocclusion edge pixels. Removed region is filled with surrounding background contents, and depth information is used in this process to prevent foreground penetration. The predicted background contents are warped to the disocclusion region, thereby achieving the hole filling. Experimental results show that the proposed method performs better than the other methods in disocclusion filling, and improves the subjective and objective quality of the synthesized view. In the evaluation results of PSNR, SSIM, FSIMc and VSI, our method improves by 0.

show abstract

Section: G Discussionmentioning

confidence: 99%

Local Foreground Removal Disocclusion Filling Method for View Synthesis

Liang

Chen

et al. 2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…A popular method in virtual view synthesis is Depth Image Based Rendering (DIBR). Using pictures shot from different viewpoints along with corresponding depth images, DIBR methods can synthesize the picture from a virtual viewpoint by 3D image warping and inpainting [7]- [10]. Another popular method is to use Neural Networks to synthesize virtual viewpoints from pure images inputs [11]- [13] or reconstruct the 3D model of the scene [14]- [17].…”

Section: Related Workmentioning

confidence: 99%

Edge-FVV: Free Viewpoint Video Streaming by Learning at the Edge

Zhang,

Feng

et al. 2023

2023 IEEE International Conference on Multimedia and Expo (ICME)

View full text Add to dashboard Cite

Audiences can gain an immersive experience watching videos from multiple angles (a.k.a. viewpoints). Free Viewpoint Video (FVV) is developed to enable users to choose their preferred viewpoints during the play of a video. However, users may experience a delay if video frames of the chosen viewpoint cannot be timely loaded, or synthesized from multiple video streams of neighboring viewpoints. To address this problem, we present Edge-FVV, an edge-assisted FVV system that employs edge caches to reduce the delay in streaming the requested FVV from the server to client users. We first analyze the capacity and delay at edge caches when answering FVV requests. Next, we propose two types of machine learning algorithms that allocate the users' requests to appropriate edge caches. Our evaluation shows that two types of proposed algorithms outperform benchmarks by 4.2-7.4% and 4.6-6.8%, respectively, in reducing the delay for FVV requests.

show abstract

“…Sun et al [23] propose an online background model construction method based on switchable Gaussian model (SGM), which improves the adaptability to the scene. Deng et al [24] establish an adaptive background model based on the Gaussian mixture model (GMM) to fill the dis-occluded regions. However, for a single reference image and still foreground objects, the occluded backgrounds in the scene may not be fully obtained, so it is difficult to establish a good background model in these situations.…”

Section: Related Workmentioning

confidence: 99%

“…This is because the background region, which is not visible in the left reference view may be visible in the right reference view, so the hole-filling results can be very accurate. Deng et al [24] also use two reference views in their method, and obtain the virtual view through a weighted view blending process. However, multi-view rendering requires more devices and transmission bandwidth compared to single view rendering.…”

Section: Related Workmentioning

confidence: 99%

Artifact Handling Based on Depth Image for View Synthesis

Chen

Liang

et al. 2019

Applied Sciences

View full text Add to dashboard Cite

The depth image based rendering (DIBR) is a popular technology for 3D video and free viewpoint video (FVV) synthesis, by which numerous virtual views can be generated from a single reference view and its depth image. However, some artifacts are produced in the DIBR process and reduce the visual quality of virtual view. Due to the diversity of artifacts, effectively handling them becomes a challenging task. In this paper, an artifact handling method based on depth image is proposed. The reference image and its depth image are extended to fill the holes that belong to the out-of-field regions. A depth image preprocessing method is applied to project the ghosts to their correct place. The 3D warping process is optimized by an adaptive one-to-four method to deal with the cracks and pixel overlapping. For disocclusions, we calculate depth and background terms of the filling priority based on depth information. The search for the best matching patch is performed simultaneously in the reference image and the virtual image. Moreover, adaptive patch size is used in all hole-filling processes. Experimental results demonstrate the effectiveness of the proposed method, which has better performance compared with previous methods in subjective and objective evaluation.

show abstract

Reliability-Based View Synthesis for Free Viewpoint Video

Cited by 8 publications

References 22 publications

Local Foreground Removal Disocclusion Filling Method for View Synthesis

Local Foreground Removal Disocclusion Filling Method for View Synthesis

Edge-FVV: Free Viewpoint Video Streaming by Learning at the Edge

Artifact Handling Based on Depth Image for View Synthesis

Contact Info

Product

Resources

About