Neural Photometry-Guided Visual Attribute Transfer

Rodríguez-Pardo, Carlos; Garcés, Elena

doi:10.1109/tvcg.2021.3133081

Cited by 8 publications

(5 citation statements)

References 81 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Albedo SVBRDF-net [26], NeRD [7], MLPs, spherical Gaussians, diffusion priors, semantic segmentation [27] Realistic image synthesis, BRDF parameter estimation, perceptual evaluation SVBRD Cascaded network architectures [28], MLPs, spherical Gaussians, autoencoders, neural textures, renderers [29], NeRF [30] SVBRDF prediction accuracy, perceptual evaluation Surface Properties…”

Section: Reflectancerelatedmentioning

confidence: 99%

“…Estimation of the Spatially Varying Bidirectional Reflectance Distribution Function (SVBRDF) involves advanced network designs for shape, illumination, and SVBRDF predictions, such as cascaded network architectures [28]. Moreover, combinations of methods like MLPs and spherical Gaussians have shown promise [33], along with neural networks incorporating autoencoders, neural textures, and renderers [29]. Disentangling scenes into meso-structure textures and underlying base shapes has enabled the estimation of diffuse and specular coefficients using neural radiance fields (NeRFs) [30].…”

Section: Materials Visual Attributes: Reflectance-relatedmentioning

confidence: 99%

See 1 more Smart Citation

Material Attribute Estimation as Part of Telecommunication Augmented Reality, Virtual Reality, and Mixed Reality System: Systematic Review

Christoff,

Tonchev

2024

Electronics

View full text Add to dashboard Cite

The integration of material attribute estimation (MAE) within augmented reality, virtual reality, and mixed reality telecommunication systems stands as a pivotal domain, evolving rapidly with the advent of the Tactile Internet. This unifying implementation process has the potential for improvements in the realism and interactivity of immersive environments. The interaction between MAE and the haptic Internet could lead to significant advances in haptic feedback systems, enabling more accurate and responsive user experiences. This systematic review is focused on the intersection of MAE and the Tactile Internet, aiming to find an implementation path between these technologies. Motivated by the potential of the haptic Internet to advance telecommunications, we explore its potential to advance the analysis of material attributes within AR, VR, and MR applications. Through an extensive analysis of current research approaches, including machine learning methods, we explore the possibilities of integrating the TI into MAE. By exploiting haptic and visual properties stored in the materials of 3D objects and using them directly during rendering in remote access scenarios, we propose a conceptual framework that combines data capture, visual representation, processing, and communication in virtual environments.

show abstract

Section: Reflectancerelatedmentioning

confidence: 99%

Section: Materials Visual Attributes: Reflectance-relatedmentioning

confidence: 99%

Material Attribute Estimation as Part of Telecommunication Augmented Reality, Virtual Reality, and Mixed Reality System: Systematic Review

Christoff,

Tonchev

2024

Electronics

View full text Add to dashboard Cite

show abstract

“…To facilitate this process, different methods were proposed to decompose SVBRDF in different components, easier to individually edit [LBAD*06, LL11]. More recently, learning‐based methods were proposed to transfer material properties [DDB20, RPG22, FR22] or interpolate and re‐sample them [HDMR21]. Our method allows to edit the input noises and patterns, as well as provide guide maps to constrain the optimization result, providing control over the generated material structure (see Figure 10).…”

Section: Related Workmentioning

confidence: 99%

A Semi‐Procedural Convolutional Material Prior

Zhou

Hašan

Deschaintre

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

Lightweight material capture methods require a material prior, defining the subspace of plausible textures within the large space of unconstrained texel grids. Previous work has either used deep neural networks (trained on large synthetic material datasets) or procedural node graphs (constructed by expert artists) as such priors. In this paper, we propose a semi‐procedural differentiable material prior that represents materials as a set of (typically procedural) grayscale noises and patterns that are processed by a sequence of lightweight learnable convolutional filter operations. We demonstrate that the restricted structure of this architecture acts as an inductive bias on the space of material appearances, allowing us to optimize the weights of the convolutions per‐material, with no need for pre‐training on a large dataset. Combined with a differentiable rendering step and a perceptual loss, we enable single‐image tileable material capture comparable with state of the art. Our approach does not target the pixel‐perfect recovery of the material, but rather uses noises and patterns as input to match the target appearance. To achieve this, it does not require complex procedural graphs, and has a much lower complexity, computational cost and storage cost. We also enable control over the results, through changing the provided patterns and using guide maps to push the material properties towards a user‐driven objective.

show abstract

“…We train 𝒞 on linear RGB space, with a pixel‐wise reconstruction loss. Specifically, we use an ℒ 1 loss, as it produces sharper and more accurate reconstructions than higher order norms, such as ℒ 2 [IZZE17,RPG21]. We also observe strong gradient instabilities when training with L 2 on linear RGB.…”

Section: Environment Map Compressionmentioning

confidence: 99%

NEnv: Neural Environment Maps for Global Illumination

Rodríguez-Pardo

Fabre

Garcés

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

Environment maps are commonly used to represent and compute far‐field illumination in virtual scenes. However, they are expensive to evaluate and sample from, limiting their applicability to real‐time rendering. Previous methods have focused on compression through spherical‐domain approximations, or on learning priors for natural, day‐light illumination. These hinder both accuracy and generality, and do not provide the probability information required for importance‐sampling Monte Carlo integration. We propose NEnv, a deep‐learning fully‐differentiable method, capable of compressing and learning to sample from a single environment map. NEnv is composed of two different neural networks: A normalizing flow, able to map samples from uniform distributions to the probability density of the illumination, also providing their corresponding probabilities; and an implicit neural representation which compresses the environment map into an efficient differentiable function. The computation time of environment samples with NEnv is two orders of magnitude less than with traditional methods. NEnv makes no assumptions regarding the content (i.e. natural illumination), thus achieving higher generality than previous learning‐based approaches. We share our implementation and a diverse dataset of trained neural environment maps, which can be easily integrated into existing rendering engines.

show abstract

Neural Photometry-Guided Visual Attribute Transfer

Cited by 8 publications

References 81 publications

Material Attribute Estimation as Part of Telecommunication Augmented Reality, Virtual Reality, and Mixed Reality System: Systematic Review

Material Attribute Estimation as Part of Telecommunication Augmented Reality, Virtual Reality, and Mixed Reality System: Systematic Review

A Semi‐Procedural Convolutional Material Prior

NEnv: Neural Environment Maps for Global Illumination

Contact Info

Product

Resources

About