MegaParallax: Casual 360° Panoramas with Motion Parallax

Bertel, Tobias; Campbell, Neill D. F.; Richardt, Christian

doi:10.1109/tvcg.2019.2898799

Cited by 40 publications

(43 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Normally, 360 • images presented in VR are stereoscopic. Thus solutions for capturing and rendering stereo imagery have been proposed with both fixed camera arrays [24][25][26][27] and casual photography [28][29][30]. The raw images and video are then warped and stitched to make a 360 • panorama [31] for VR display.…”

Section: Panoramic Image and Video Creationmentioning

confidence: 99%

VR content creation and exploration with deep learning: A survey

Wang

Lyu

et al. 2020

Comp. Visual Media

View full text Add to dashboard Cite

Virtual reality (VR) offers an artificial, computer generated simulation of a real life environment. It originated in the 1960s and has evolved to provide increasing immersion, interactivity, imagination, and intelligence. Because deep learning systems are able to represent and compose information at various levels in a deep hierarchical fashion, they can build very powerful models which leverage large quantities of visual media data. Intelligence of VR methods and applications has been significantly boosted by the recent developments in deep learning techniques. VR content creation and exploration relates to image and video analysis, synthesis and editing, so deep learning methods such as fully convolutional networks and general adversarial networks are widely employed, designed specifically to handle panoramic images and video and virtual 3D scenes. This article surveys recent research that uses such deep learning methods for VR content creation and exploration. It considers the problems involved, and discusses possible future directions in this active and emerging research area. Keywords virtual reality; deep learning; neural networks; 360 • image and video virtual content

show abstract

Section: Panoramic Image and Video Creationmentioning

confidence: 99%

VR content creation and exploration with deep learning: A survey

Wang

Lyu

et al. 2020

Comp. Visual Media

View full text Add to dashboard Cite

show abstract

“…Furthermore, the photos have often been taken at different points in time, making 3D reconstruction problematic as the illumination can differ significantly and objects may have moved in the scene [39]. Finally, another interesting capture strategy is unstructured video capture, where a user records video while moving the camera to capture multiple viewpoints [40,41,42]. In this thesis, we employ unstructured capture strategies that can easily be performed by just one camera operator, and also provide footage suitable for VR experiences with 360°coverage of the scene.…”

Section: Unstructured Capturementioning

confidence: 99%

“…However, these approaches focus on object capture, resulting in small light fields with a limited field of view. Recently, image warping has made it possible to build larger, panoramic light fields from hand-held footage, although the output camera motion is restricted to a disk [42]. Even with geometry from modern 3D reconstruction techniques, motorized capture rigs are still necessary to build light fields with a large enough range of motion for seated VR experiences [29].…”

Section: Light Fieldsmentioning

confidence: 99%

Viewpoint-Free Photography for Virtual Reality

Hedman¹

2020

Real VR – Immersive Digital Reality

View full text Add to dashboard Cite

Viewpoint-free photography, i.e., interactively controlling the viewpoint of a photograph after capture, is a standing challenge. In this thesis, we investigate algorithms to enable viewpoint-free photography for virtual reality (VR) from casual capture, i.e., from footage easily captured with consumer cameras. We build on an extensive body of work in image-based rendering (IBR). Given images of an object or scene, IBR methods aim to predict the appearance of an image taken from a novel perspective. Most IBR methods focus on full or near-interpolation, where the output viewpoints either lie directly between captured images, or nearby. These methods are not suitable for VR, where the user has significant range of motion and can look in all directions. Thus, it is essential to create viewpoint-free photos with a wide field-of-view and sufficient positional freedom to cover the range of motion a user might experience in VR. We focus on two VR experiences: 1) Seated VR experiences, where the user can lean in different directions. This simplifies the problem, as the scene is only observed from a small range of viewpoints. Thus, we focus on easy capture, showing how to turn panorama-style capture into 3D photos, a simple representation for viewpoint-free photos, and also how to speed up processing so users can see the final result on-site. 2) Room-scale VR experiences, where the user can explore vastly different perspectives. This is challenging: More input footage is needed, maintaining real-time display rates becomes difficult, view-dependent appearance and object backsides need to be modelled, all while preventing noticeable mistakes. We address these challenges by: (1) creating refined geometry for each input photograph, (2) using a fast tiled rendering algorithm to achieve real-time display rates, and (3) using a convolutional neural network to hide visual mistakes during compositing.

show abstract

“…To overcome these limitations, some systems use geometric proxies or view interpolation techniques originally developed for imagebased rendering systems. While the earliest lightfield capture and rendering systems such as the Lumigraph [Gortler et al 1996] and Unstructured Lumigraph [Buehler et al 2001] used global 3D mesh proxies, more recent systems such as [Hedman et al 2016;Overbeck et al 2018;Penner and Zhang 2017;Zitnick et al 2004] estimate per-camera depth maps or pair-wise flow fields [Bertel et al 2019] and use these to warp the source images into the desired novel viewpoints. Other approaches focus on adaptive meshing and mesh tracking to create temporally coherent geometry [Collet et al 2015].…”

Section: Related Workmentioning

confidence: 99%

“…To achieve high rendering quality, many of these systems use a large number of images and cameras swept on an circular arc or spherical surface [Bertel et al 2019;Overbeck et al 2018], which restricts their use to still scenes. For example, the system built by [Overbeck et al 2018] takes between 30 seconds and 33 minutes to acquire a 360 • scene.…”

Section: Related Workmentioning

confidence: 99%

An integrated 6DoF video camera and system design

et al. 2019

View full text Add to dashboard Cite

Fig. 1. The commercial 16 camera system, an equirectangular depth map, and final color rendering produced from our system. Designing a fully integrated 360 • video camera supporting 6DoF head motion parallax requires overcoming many technical hurdles, including camera placement, optical design, sensor resolution, system calibration, real-time video capture, depth reconstruction, and real-time novel view synthesis. While there is a large body of work describing various system components, such as multi-view depth estimation, our paper is the first to describe a complete, reproducible system that considers the challenges arising when designing, building, and deploying a full end-to-end 6DoF video camera and playback environment. Our system includes a computational imaging software pipeline supporting online markerless calibration, high-quality reconstruction, and real-time streaming and rendering. Most of our exposition is based on a professional 16-camera configuration, which will be commercially available to film producers. However, our software pipeline is generic and can handle a variety of camera geometries and configurations. The entire calibration and reconstruction software pipeline along with example datasets is open sourced to encourage follow-up research in high-quality 6DoF video reconstruction and rendering 1 .

show abstract

MegaParallax: Casual 360° Panoramas with Motion Parallax

Cited by 40 publications

References 37 publications

VR content creation and exploration with deep learning: A survey

VR content creation and exploration with deep learning: A survey

Viewpoint-Free Photography for Virtual Reality

An integrated 6DoF video camera and system design

Contact Info

Product

Resources

About