AR-ia: Volumetric Opera for Mobile Augmented Reality

Kelly, Seán; Cordingley, Samantha; Nolan, Patrick; Rhemann, Christoph; Fanello, Sean; Tang, Dajun; Osborn, Jude; Busch, Jay; Davidson, Philip; Debevec, Paul; Denny, Peter; Fyffe, Graham; Guo, Kaiwen; Harvey, Geoff; Izadi, Shahram; Lincoln, Peter; Alex, Wan-Chun; Taylor, Jonathan; Yu, Xing; Whalen, Matt; Dourgarian, Jason; Blanchett, Genevieve; French, Narelle; Sillitoe, Kirstin; Uglow, Tea; Spiteri, Brenton; Pearson, Emma; Kernot, Wade; Richards, Jonathan

doi:10.1145/3355355.3361890

Cited by 3 publications

(52 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Synthesizing realistic relightable humans it is often tackled using image-based relighting techniques [Debevec et al 2000;Einarsson et al 2006;Wenger et al 2005b], or Parametric models with priors [Barron and Malik 2015;Blanz and Vetter 1999;Garrido et al 2013Garrido et al , 2016Gotardo et al 2018;Ichim et al 2015;Meka et al 2017;Pons-Moll et al 2015;Theobalt et al 2007;Thies et al 2016; Wen et al Table 1. Machine learning methods achieve a high degree of photorealism, whereas traditional capture pipelines [Guo et al 2019] are better at generalization, rendering capabilities, and they can capture moving performers. Our algorithm brings together the capabilities of both state-of-the-art approaches.…”

Section: Related Workmentioning

confidence: 99%

“…Relightable Moving Performers Higher-order Appearance Model Generalization Guo et al [2019] Yes Yes Yes No Yes Thies et al [2019] Yes No No Yes No Martin-Brualla et al [2018] Yes No Yes Yes Yes Wenger et al [2005a] No Yes Yes (with high FPS) Yes Yes Meka et al [2019] No Yes Yes Yes Yes Yes (half hemisphere) No…”

Section: Free Viewpointmentioning

confidence: 99%

“…Capturing and rendering photorealistic human performances with controllable viewpoint and lighting is one of the most active research areas in the fields of computer graphics and vision. Such a technology has applications in augmented and virtual reality [Kelly et al 2019;Lombardi et al 2018;, movie production [Seymour 2020], and game development.…”

Section: Introductionmentioning

confidence: 99%

“…Capturing and rendering a large number of performers in a photorealistic manner with no manual intervention is still a very active research area. Recently, [Guo et al 2019] built a fully distributed pipeline for volumetric performance capture that combines volumetric capture with relighting based on time-multiplexed color gradient illuminations. These alternating lighting conditions have been shown to enable the estimation of the material properties such as albedo, roughness, and surface normals [Fyffe et al 2009].…”

Section: Introductionmentioning

confidence: 99%

“…Despite impressive results and the evident quality improvement over previous capture solutions, such a system still lacks photorealism. Although the 3D capture pipeline uses custom high quality depth sensors [Guo et al 2019;, it cannot precisely reconstruct thin complex structures such as hair and clothing. The reflectance estimation proposed in [Guo et al 2019] relies on a simplistic cosine lobe reflectance model [Fyffe et al 2009] and therefore cannot accurately model higher-frequency view-dependent specular effects or the complex luster of cloth and skin.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Deep relightable textures

et al. 2020

Self Cite

View full text Add to dashboard Cite

under desired viewpoint and lighting. Our system leverages deep neural networks that model the classical rendering process to learn implicit features that represent the view-dependent appearance of the subject independent of the geometry layout, allowing for generalization to unseen subject poses and even novel subject identity. Detailed experiments and comparisons demonstrate the efficacy and versatility of our method to generate high-quality results, significantly outperforming the existing state-of-the-art solutions. CCS Concepts: • Computing methodologies → Computer vision; Machine learning; Rendering.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Free Viewpointmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Deep relightable textures

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

Data analysis of music preferences of web users based on social and demographic factors

Shakhovska

Федушко

2022

Procedia Computer Science

View full text Add to dashboard Cite

Neural Light Transport for Relighting and View Synthesis

et al. 2021

Self Cite

View full text Add to dashboard Cite

The light transport (LT) of a scene describes how it appears under different lighting conditions from different viewing directions, and complete knowledge of a scene’s LT enables the synthesis of novel views under arbitrary lighting. In this article, we focus on image-based LT acquisition, primarily for human bodies within a light stage setup. We propose a semi-parametric approach for learning a neural representation of the LT that is embedded in a texture atlas of known but possibly rough geometry. We model all non-diffuse and global LT as residuals added to a physically based diffuse base rendering. In particular, we show how to fuse previously seen observations of illuminants and views to synthesize a new image of the same scene under a desired lighting condition from a chosen viewpoint. This strategy allows the network to learn complex material effects (such as subsurface scattering) and global illumination (such as diffuse interreflection), while guaranteeing the physical correctness of the diffuse LT (such as hard shadows). With this learned LT, one can relight the scene photorealistically with a directional light or an HDRI map, synthesize novel views with view-dependent effects, or do both simultaneously, all in a unified framework using a set of sparse observations. Qualitative and quantitative experiments demonstrate that our Neural Light Transport (NLT) outperforms state-of-the-art solutions for relighting and view synthesis, without requiring separate treatments for both problems that prior work requires. The code and data are available at http://nlt.csail.mit.edu.

show abstract

AR-ia: Volumetric Opera for Mobile Augmented Reality

Cited by 3 publications

References 10 publications

Deep relightable textures

Deep relightable textures

Data analysis of music preferences of web users based on social and demographic factors

Neural Light Transport for Relighting and View Synthesis

Contact Info

Product

Resources

About