A system for acquiring, processing, and rendering panoramic light field stills for virtual reality

Overbeck, Ryan S.; Erickson, Daniel; Evangelakos, Daniel; Pharr, Matt; Debevec, Paul

doi:10.1145/3272127.3275031

Cited by 139 publications

(121 citation statements)

References 45 publications

Supporting

Mentioning

108

Contrasting

Order By: Relevance

“…Rendering algorithms based on plenoptic sampling enjoy the significant benefit of prescriptive sampling; given a new scene, it is easy to compute the required view sampling density to enable highquality renderings. Many modern light field acquisition systems have been designed based on these principles, including large-scale camera systems [Overbeck et al 2018;Wilburn et al 2005] and a mobile phone app [Davis et al 2012].…”

Section: Plenoptic Sampling and Reconstructionmentioning

confidence: 99%

Local light field fusion

Mildenhall

Srinivasan

Ortiz-Cayon³

et al. 2019

ACM Trans. Graph.

786

126

View full text Add to dashboard Cite

show abstract

Section: Plenoptic Sampling and Reconstructionmentioning

confidence: 99%

Local light field fusion

Mildenhall

Srinivasan

Ortiz-Cayon³

et al. 2019

ACM Trans. Graph.

786

126

View full text Add to dashboard Cite

show abstract

“…However, it is rarely practical to measure a densely sampled light field. Instead, scenes are most often recorded using a limited number of viewpoints that sparsely sample the light field [20]. View synthesis research has focused on developing priors that aid in recovery of a dense light field from such sparse measurements.…”

Section: Background and Related Workmentioning

confidence: 99%

DeepView: View Synthesis With Learned Gradient Descent

Flynn

Broxton

Debevec

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

437

286

View full text Add to dashboard Cite

Figure 1: The DeepView architecture. (a) The network takes a sparse set of input images shot from different viewpoints. (b, c) The scene is reconstructed using learned gradient descent, producing a multi-plane image (a series of fronto-parallel, RGBA textured planes). (d) The multi-plane image is suitable for real-time, high-quality rendering of novel viewpoints. The result above uses four input views in a 30cm × 20cm rectangular layout. The novel view was rendered with a virtual camera positioned at the centroid of the four input views. More results, including video and an interactive viewer, at: https://augmentedperception.github.io/deepview/ AbstractWe present a novel approach to view synthesis using multiplane images (MPIs). Building on recent advances in learned gradient descent, our algorithm generates an MPI from a set of sparse camera viewpoints. The resulting method incorporates occlusion reasoning, improving performance on challenging scene features such as object boundaries, lighting reflections, thin structures, and scenes with high depth complexity. We show that our method achieves high-quality, state-of-the-art results on two datasets: the Kalantari light field dataset, and a new camera array dataset, Spaces, which we make publicly available.

show abstract

“…Indeed, it is possible to create very high quality viewpoint-free photographs by exhaustively capturing footage with a motorized-multi camera rig [29]. This is well suited for seated VR experiences with complete 360°coverage, but a limited range of motion.…”

Section: Rig Capturementioning

confidence: 99%

“…This also imposes requirements on the earlier stages of viewpoint-free photography pipelines. For example, a representation which relies on many input photos needs a convenient capture strategy and a scalable approach to sparse reconstruction [29]. Alternatively, representations that require high quality geometry rely heavily on the dense reconstruction stage [101].…”

Section: Representations For Renderingmentioning

confidence: 99%

Viewpoint-Free Photography for Virtual Reality

Hedman¹

2020

Real VR – Immersive Digital Reality

View full text Add to dashboard Cite

Viewpoint-free photography, i.e., interactively controlling the viewpoint of a photograph after capture, is a standing challenge. In this thesis, we investigate algorithms to enable viewpoint-free photography for virtual reality (VR) from casual capture, i.e., from footage easily captured with consumer cameras. We build on an extensive body of work in image-based rendering (IBR). Given images of an object or scene, IBR methods aim to predict the appearance of an image taken from a novel perspective. Most IBR methods focus on full or near-interpolation, where the output viewpoints either lie directly between captured images, or nearby. These methods are not suitable for VR, where the user has significant range of motion and can look in all directions. Thus, it is essential to create viewpoint-free photos with a wide field-of-view and sufficient positional freedom to cover the range of motion a user might experience in VR. We focus on two VR experiences: 1) Seated VR experiences, where the user can lean in different directions. This simplifies the problem, as the scene is only observed from a small range of viewpoints. Thus, we focus on easy capture, showing how to turn panorama-style capture into 3D photos, a simple representation for viewpoint-free photos, and also how to speed up processing so users can see the final result on-site. 2) Room-scale VR experiences, where the user can explore vastly different perspectives. This is challenging: More input footage is needed, maintaining real-time display rates becomes difficult, view-dependent appearance and object backsides need to be modelled, all while preventing noticeable mistakes. We address these challenges by: (1) creating refined geometry for each input photograph, (2) using a fast tiled rendering algorithm to achieve real-time display rates, and (3) using a convolutional neural network to hide visual mistakes during compositing.

show abstract

A system for acquiring, processing, and rendering panoramic light field stills for virtual reality

Abstract: generating stereo views at 90Hz on commodity VR hardware. Using our system, we built a freely available light field experience application called Welcome to Light Fields featuring a library of panoramic light field stills for consumer VR which has been downloaded over 15,000 times.

Cited by 139 publications

References 45 publications

Local light field fusion

Local light field fusion

DeepView: View Synthesis With Learned Gradient Descent

Viewpoint-Free Photography for Virtual Reality

Contact Info

Product

Resources

About