Depth Image-Based Rendering of Non-Lambertian Content in MPEG Immersive Video

Fachada, Sarah; Bonatto, Daniele; Xie, Yupeng; Alface, Patrice Rondão; Teratani, Mehrdad; Lafruit, Gauthier

doi:10.1109/ic3d53758.2021.9687263

Cited by 5 publications

(5 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The corresponding metadata should be efficiently compressed and added to the MIV bitstream to assist the renderer in reconstructing a novel viewport of the scene in a photorealistic manner. Some work [26] based on extending depth-image-based rendering has already been started as an exploration experiment.…”

Section: Non-lambertian Scenesmentioning

confidence: 99%

The MPEG Immersive Video Standard—Current Status and Future Outlook

Vadakital¹,

Dziembowski

Lafruit

et al. 2022

IEEE MultiMedia

Self Cite

View full text Add to dashboard Cite

The MPEG immersive video (MIV) standard is the latest addition to the MPEG-I suite of standards.It focuses on the representation and coding of immersive media. MIV is designed to support virtual and extended reality (VR/XR) applications that require six degrees of freedom (6DoF) visual interaction with the rendered scene. Edition-1 of MIV is now in its final phase of standardization. Leveraging conventional 2D video codecs, the MIV standard efficiently codes volumetric scenes and allows advanced visual effects like bullet-time flythroughs. The video feeds capturing the scene are first processed to identify a set of basic views that are augmented with additional information from all other views. The data is then intelligently packed into atlases and further compressed with any existing 2D video codec of choice. Experimental results show BD-PSNR gains of up to 6 dB in the 10 to 20 Mbps range compared to a naive simulcast multiview video coding approach. The paper concludes with an outlook on future extensions for the second edition of MIV.

show abstract

Section: Non-lambertian Scenesmentioning

confidence: 99%

The MPEG Immersive Video Standard—Current Status and Future Outlook

Vadakital¹,

Dziembowski

Lafruit

et al. 2022

IEEE MultiMedia

Self Cite

View full text Add to dashboard Cite

show abstract

“…RVS takes as input static or dynamic content [2] and has been used in a variety of contexts, including Super-MultiView light field displays [19], foveated light field displays [20] and engraved holographic stereograms [21]. The last version (v4.0) adds basic support for non-Lambertian content [22,23] (i.e. objects with transparency and reflections) by enhancing the depth maps with polynomial interpolation techniques.…”

Section: Related Workmentioning

confidence: 99%

Novel view synthesis in embedded virtual reality devices

Laurie¹,

Daniele²,

Sarah³

et al. 2022

Self Cite

View full text Add to dashboard Cite

Virtual Reality and Free Viewpoint navigation require highquality rendered images to be realistic. Current hardware assisted raytracing methods cannot reach the expected quality in real-time and are also limited by the 3D mesh quality. An alternative is depth image-based rendering (DIBR) where the input only consists of images and their associated depth maps for synthesizing virtual views to the Head Mounted Display (HMD). The MPEG Immersive Video (MIV) standard uses such a DIBR algorithm called the Reference View Synthesizer (RVS). We have first implemented a GPU version, called the Real-time Accelerated View Synthesizer (RaViS), that synthesizes two virtual views in real-time for the HMD. In the present paper, we explore the differences between desktop and embedded GPU platforms, porting RaViS to an embedded HMD without the need for a separate, discrete desktop GPU. The proposed solution gives a first insight into DIBR View Synthesis techniques in embedded HMDs using OpenGL and Vulkan, a cross-platform 3D rendering library with support for embedded devices.

show abstract

“…Several papers regard the atlas preparation step, including the patch merging [12], allowing for more efficient packing of non-pruned information within atlases, non-linear depth quantization [13], or adaptation of patch occupancy [14] to the block-based nature of typical video encoders, such as HEVC or VVC. Other works focus on rendering and increasing the quality of synthesized viewports for immersive video [15] and non-Lambertian content [16].…”

Section: Related Workmentioning

confidence: 99%

Advanced HEVC Screen Content Coding for MPEG Immersive Video

2022

View full text Add to dashboard Cite

This paper presents the modified HEVC Screen Content Coding (SCC) that was adapted to be more efficient as an internal video coding of the MPEG Immersive Video (MIV) codec. The basic, unmodified SCC is already known to be useful in such an application. However, in this paper, we propose three additional improvements to SCC to increase the efficiency of immersive video coding. First, we analyze using the quarter-pel accuracy in the intra block copy technique to provide a more effective search of the best candidate block to be copied in the encoding process. The second proposal is the use of tiles to allow inter-view prediction inside MIV atlases. The last proposed improvement is the addition of the MIV bitstream parser in the HEVC encoder that enables selecting the most efficient coding configuration depending on the type of currently encoded data. The experimental results show that the proposal increases the compression efficiency for natural content sequences by almost 7% and simultaneously decreases the computational time of encoding by more than 15%, making the proposal very valuable for further research on immersive video coding.

show abstract

Depth Image-Based Rendering of Non-Lambertian Content in MPEG Immersive Video

Abstract: a) Cadillac (b) Mirror (c) Magritte Mirror (d) Magritte Transparent Fig. 1: The four test sequences used in the exploration experiment. (a) (b) Courtesy of InterDigital.

Cited by 5 publications

References 20 publications

The MPEG Immersive Video Standard—Current Status and Future Outlook

The MPEG Immersive Video Standard—Current Status and Future Outlook

Novel view synthesis in embedded virtual reality devices

Advanced HEVC Screen Content Coding for MPEG Immersive Video

Contact Info

Product

Resources

About