Local light field fusion

Mildenhall, Ben; Srinivasan, Pratul P.; Ortiz-Cayon, Rodrigo; Kalantari, Nima Khademi; Ramamoorthi, Ravi; Ng, Ren; Kar, Abhishek

doi:10.1145/3306346.3322980

Cited by 787 publications

(206 citation statements)

References 49 publications

Supporting

Mentioning

205

Contrasting

Order By: Relevance

“…A good baseline for image-based rendering is the unstructured lumigraph rendering (ULR) algorithm [2]: it efficiently demonstrates the potential of view-dependent rendering algorithms and has consistently been used by recent works as a reference point for comparison [1,10,11,15]. Therefore, we decided to implement our own version of ULR using vertex/fragment shaders.…”

Section: View-dependent Rendering Of a Global Mesh 311 Our Implemenmentioning

confidence: 99%

“…of weights thus have to be passed in the interpolators. Finally, unstructured lumigraph rendering typically suffers from noticeable ghosting artifacts [15,23], linked to inconsistencies between the image data and the reconstructed camera setup and 3D mesh. This remains a limitation in our current implementation.…”

Section: Obstacles and Limitationsmentioning

confidence: 99%

“…However, there appears to currently be no easy way for smallscale content creation teams to bring the real world into VR with levels of visual accuracy similar to what is typically demonstrated by the specialized research community. Several works in the field of computer graphics have indeed demonstrated advanced methods for blending photographs together to create seamless, high-resolution virtual environments [8,15,17,23], a notable example being the publicly-released Welcome to Light Fields VR experience [17]. Unfortunately, the rendering solutions used to create such demonstrations are seldom released for public use, preventing non-expert content creators from applying these methods to their own photograph datasets.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

From Real to Virtual: An Image-Based Rendering Toolkit to Help Bring the World Around Us Into Virtual Reality

Dinechin¹,

Paljic²

2020

2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)

View full text Add to dashboard Cite

The release of consumer-grade head-mounted displays has helped bring virtual reality (VR) to our homes, cultural sites, and workplaces, increasingly making it a part of our everyday lives. In response, many content creators have expressed renewed interest in bringing the people, objects, and places of our daily lives into VR, helping push the boundaries of our ability to transform photographs of everyday real-world scenes into convincing VR assets. In this paper, we present an open-source solution we developed in the Unity game engine as a way to make this image-based approach to virtual reality simple and accessible to all, to encourage content creators of all kinds to capture and render the world around them in VR. We start by presenting the use cases of image-based virtual reality, from which we discuss the motivations that led us to work on our solution. We then provide details on the development of the toolkit, specifically discussing our implementation of several image-based rendering (IBR) methods. Finally, we present the results of a preliminary user study focused on interface usability and rendering quality, and discuss paths for future work.

show abstract

Section: View-dependent Rendering Of a Global Mesh 311 Our Implemenmentioning

confidence: 99%

Section: Obstacles and Limitationsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

From Real to Virtual: An Image-Based Rendering Toolkit to Help Bring the World Around Us Into Virtual Reality

Dinechin¹,

Paljic²

2020

2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)

View full text Add to dashboard Cite

show abstract

“…Deep neural networks [Bengio et al 2012;Krizhevsky et al 2012;LeCun et al 2015] have shown remarkable achievements in encoding highly complex functions in a non-linear fashion. Recent works [Lombardi et al 2019;Mildenhall et al 2019;Sitzmann et al 2019] used neural networks to learn implicit representations from imagery without requiring any explicit geometry. Simple MLP-based architectures have also shown remarkable improvements in both view synthesis and in encoding complex light interactions [Kallweit et al 2017].…”

Section: Related Workmentioning

confidence: 99%

Neural light field 3D printing

et al. 2020

View full text Add to dashboard Cite

our approach brings significant visual quality improvement compared to the multilayer and uniform grid-based approaches. We validate our simulations with fabricated prototypes and demonstrate that our pipeline is flexible enough to allow fabrications of both planar and non-planar displays. CCS Concepts: • Applied computing → Computer-aided manufacturing; • Computing methodologies → Neural networks; Volumetric models.

show abstract

“…While our reconstruction pipeline is based on traditional coarseto-fine, hierarchical patch matching, one could also apply one of the more recent machine learning-based approaches such as [Chabra et al 2019;Donne and Geiger 2019;Mildenhall et al 2019;Tonioni et al 2019;Xu et al 2019;Yao et al 2019b;Zhang et al 2019]. Since we are open sourcing our multi-view stereo datasets, we hope that others will investigate whether further improvements could be obtained using these newer approaches.…”

Section: D Reconstruction / Depth Estimationmentioning

confidence: 99%

An integrated 6DoF video camera and system design

et al. 2019

View full text Add to dashboard Cite

Fig. 1. The commercial 16 camera system, an equirectangular depth map, and final color rendering produced from our system. Designing a fully integrated 360 • video camera supporting 6DoF head motion parallax requires overcoming many technical hurdles, including camera placement, optical design, sensor resolution, system calibration, real-time video capture, depth reconstruction, and real-time novel view synthesis. While there is a large body of work describing various system components, such as multi-view depth estimation, our paper is the first to describe a complete, reproducible system that considers the challenges arising when designing, building, and deploying a full end-to-end 6DoF video camera and playback environment. Our system includes a computational imaging software pipeline supporting online markerless calibration, high-quality reconstruction, and real-time streaming and rendering. Most of our exposition is based on a professional 16-camera configuration, which will be commercially available to film producers. However, our software pipeline is generic and can handle a variety of camera geometries and configurations. The entire calibration and reconstruction software pipeline along with example datasets is open sourced to encourage follow-up research in high-quality 6DoF video reconstruction and rendering 1 .

show abstract

Local light field fusion

Abstract: demonstrate our approach's practicality with an augmented reality smartphone app that guides users to capture input images of a scene and viewers that enable realtime virtual exploration on desktop and mobile platforms.

Cited by 787 publications

References 49 publications

From Real to Virtual: An Image-Based Rendering Toolkit to Help Bring the World Around Us Into Virtual Reality

From Real to Virtual: An Image-Based Rendering Toolkit to Help Bring the World Around Us Into Virtual Reality

Neural light field 3D printing

An integrated 6DoF video camera and system design

Contact Info

Product

Resources

About