NeRF‐Tex: Neural Reflectance Field Textures

Baatz, H.; Granskog, Jonathan; Papas, Marios; Rousselle, Fabrice; Novák, Jan

doi:10.1111/cgf.14449

Cited by 29 publications

(24 citation statements)

References 89 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In future work, we believe our Active Exploration approach has significant promise for any neural rendering method (e.g., [Baatz et al 2021]) that trains on synthetic data, allowing potentially significant reduction in training time and improvements in quality.…”

Section: Ground Truthmentioning

confidence: 99%

“…Neural rendering is an exciting emerging alternative to traditional physically-based rendering; Initial attempts [Ren et al 2013] were restricted to indirect light with static geometry, while more recent methods [Eslami et al 2018;Granskog et al 2020] are trained on large numbers of rendered images for variable scenes, i.e., with changing objects, materials, lights and viewpoint. Other recent solutions learn encodings of appearance or lighting [Baatz et al 2021;Nalbach et al 2017;Zhu et al 2021], again training on rendered images. Uniformly sampling the space of these path-traced images is expensive; in the case of a high-dimensional space D containing all the possible configurations of a variable scene, it quickly becomes unmanageable.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Active Exploration for Neural Global Illumination of Variable Scenes

2022

View full text Add to dashboard Cite

Neural rendering algorithms introduce a fundamentally new approach for photorealistic rendering, typically by learning a neural representation of illumination on large numbers of ground truth images. When training for a given variable scene, i.e., changing objects, materials, lights and viewpoint, the space \(\mathcal {D} \) of possible training data instances quickly becomes unmanageable as the dimensions of variable parameters increase. We introduce a novel Active Exploration method using Markov Chain Monte Carlo, which explores \(\mathcal {D} \) , generating samples (i.e., ground truth renderings) that best help training and interleaves training and on-the-fly sample data generation. We introduce a self-tuning sample reuse strategy to minimize the expensive step of rendering training samples. We apply our approach on a neural generator that learns to render novel scene instances given an explicit parameterization of the scene configuration. Our results show that Active Exploration trains our network much more efficiently than uniformly sampling, and together with our resolution enhancement approach, achieves better quality than uniform sampling at convergence. Our method allows interactive rendering of hard light transport paths (e.g., complex caustics) – that require very high samples counts to be captured – and provides dynamic scene navigation and manipulation, after training for 5-18 hours depending on required quality and variations.

show abstract

Section: Ground Truthmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Active Exploration for Neural Global Illumination of Variable Scenes

2022

View full text Add to dashboard Cite

show abstract

“…Neural Fields on Manifolds. Prior works [32,11,2,57,34,30,59,28,18] use neural fields to represent a wide variety of quantities on manifolds. Oechsle et al [32] use the extrinsic embedding of the manifold to learn textures as multilayer perceptrons.…”

Section: Related Workmentioning

confidence: 99%

“…We compare with an adapted version of their method that utilizes the known geometry of the object. Baatz et al [2] introduce NeRF-Tex, a combination of neural radiance fields (NeRFs) and classical texture maps. Their method uses multiple smallscale NeRFs to cover the surface of a shape and represent mesoscale structures, such as fur, fabric, and grass.…”

Section: Related Workmentioning

confidence: 99%

“…They showed that neural radiance fields together with differentiable volume rendering can be used to reconstruct scenes and often yield photorealistic renderings from novel viewpoints. This inspired work in related fields, e.g., human shape modeling [34], shape and texture generation from text [28], and texture representation on shapes [32,2], where neural fields are able to generate a wide variety of functions with high fidelity. Fig.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Intrinsic Neural Fields: Learning Functions on Manifolds

Koestler¹,

Grittner²,

Moeller³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural fields have gained significant attention in the computer vision community due to their excellent performance in novel view synthesis, geometry reconstruction, and generative modeling. Some of their advantages are a sound theoretic foundation and an easy implementation in current deep learning frameworks. While neural fields have been applied to signals on manifolds, e.g., for texture reconstruction, their representation has been limited to extrinsically embedding the shape into Euclidean space. The extrinsic embedding ignores known intrinsic manifold properties and is inflexible wrt. transfer of the learned function. To overcome these limitations, this work introduces intrinsic neural fields, a novel and versatile representation for neural fields on manifolds. Intrinsic neural fields combine the advantages of neural fields with the spectral properties of the Laplace-Beltrami operator. We show theoretically that intrinsic neural fields inherit many desirable properties of the extrinsic neural field framework but exhibit additional intrinsic qualities, like isometry invariance. In experiments, we show intrinsic neural fields can reconstruct high-fidelity textures from images with state-of-the-art quality and are robust to the discretization of the underlying manifold. We demonstrate the versatility of intrinsic neural fields by tackling various applications: texture transfer between deformed shapes & different shapes, texture reconstruction from real-world images with view dependence, and discretization-agnostic learning on meshes and point clouds.

show abstract