Depth Augmented Omnidirectional Stereo for 6-DoF VR Photography

Bertel, Tobias; Mühlhausen, Moritz; Kappel, Moritz; Bittner, Paul Maximilian; Richardt, Christian; Magnor, Marcus

doi:10.1109/vrw50115.2020.00181

Cited by 5 publications

(2 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Techniques following this line utilize a densely sampled scene which usually requires specialized recording equipment or setups. In contrast, pointcloud representations can be obtained from arbitrary camera setups [TBLG16, TLWG17, BMK*20]. They can also easily be rendered but may suffer from artefacts introduced by disoclusions in the recorded scenes which appear as holes in the pointclouds.…”

Section: Related Workmentioning

confidence: 99%

Immersive Free‐Viewpoint Panorama Rendering from Omnidirectional Stereo Video

Mühlhausen

Kappel

Kassubeck

et al. 2023

Computer Graphics Forum

Self Cite

View full text Add to dashboard Cite

In this paper, we tackle the challenging problem of rendering real‐world 360° panorama videos that support full 6 degrees‐of‐freedom (DoF) head motion from a prerecorded omnidirectional stereo (ODS) video. In contrast to recent approaches that create novel views for individual panorama frames, we introduce a video‐specific temporally‐consistent multi‐sphere image (MSI) scene representation. Given a conventional ODS video, we first extract information by estimating framewise descriptive feature maps. Then, we optimize the global MSI model using theory from recent research on neural radiance fields. Instead of a continuous scene function, this multi‐sphere image (MSI) representation depicts colour and density information only for a discrete set of concentric spheres. To further improve the temporal consistency of our results, we apply an ancillary refinement step which optimizes the temporal coherency between successive video frames. Direct comparisons to recent baseline approaches show that our global MSI optimization yields superior performance in terms of visual quality. Our code and data will be made publicly available.

show abstract

Section: Related Workmentioning

confidence: 99%

Immersive Free‐Viewpoint Panorama Rendering from Omnidirectional Stereo Video

Mühlhausen

Kappel

Kassubeck

et al. 2023

Computer Graphics Forum

Self Cite

View full text Add to dashboard Cite

show abstract

“…As we aim to add motion parallax to arbitrary ODS videos, the sampling rate is insufficient for applying light field rendering. Pointcloud representations [3,32,34] can be rendered straightforwardly utilizing the standard rendering pipeline, but suffer from holes due to disocclusion. Accordingly easy to render are textured mesh-based Figure 3: An example for synthesizing novel viewpoints with our interleaved layer rendering approach.…”

Section: Related Workmentioning

confidence: 99%

Temporal Consistent Motion Parallax for Omnidirectional Stereo Panorama Video

Mühlhausen

Kappel

Kassubeck

et al. 2020

26th ACM Symposium on Virtual Reality Software and Technology

Self Cite

View full text Add to dashboard Cite

We present a new pipeline to enable head-motion parallax in omnidirectional stereo (ODS) panorama video rendering using a neural depth decoder. While recent ODS panorama cameras record short-baseline horizontal stereo parallax to offer the impression of binocular depth, they do not support the necessary translational degrees-of-freedom (DoF) to also provide for head-motion parallax in virtual reality (VR) applications. To overcome this limitation, we propose a pipeline that enhances the classical ODS panorama format with 6 DoF free-viewpoint rendering by decomposing the scene into a multi-layer mesh representation. Given a spherical stereo panorama video, we use the horizontal disparity to store explicit depth information for both eyes in a simple neural decoder architecture. While this approach produces reasonable results for individual frames, video rendering usually suffers from temporal depth inconsistencies. Thus, we perform successive optimization to improve temporal consistency by fine-tuning our depth decoder for both temporal and spatial smoothness. Using a consumer-grade ODS camera, we evaluate our approach on a number of real-world scene recordings and demonstrate the versatility and robustness of the proposed pipeline.

show abstract