Active Exploration for Neural Global Illumination of Variable Scenes

Abstract

Neural rendering algorithms introduce a fundamentally new approach for photorealistic rendering, typically by learning a neural representation of illumination on large numbers of ground truth images. When training for a given variable scene, i.e., changing objects, materials, lights and viewpoint, the space \(\mathcal {D} \) of possible training data instances quickly becomes unmanageable as the dimensions of variable parameters increase. We introduce a novel Active Exploration method using Markov Chain Monte Carlo, which explores \(\mathcal {D} \) , generating samples (i.e., ground truth renderings) that best help training and interleaves training and on-the-fly sample data generation. We introduce a self-tuning sample reuse strategy to minimize the expensive step of rendering training samples. We apply our approach on a neural generator that learns to render novel scene instances given an explicit parameterization of the scene configuration. Our results show that Active Exploration trains our network much more efficiently than uniformly sampling, and together with our resolution enhancement approach, achieves better quality than uniform sampling at convergence. Our method allows interactive rendering of hard light transport paths (e.g., complex caustics) – that require very high samples counts to be captured – and provides dynamic scene navigation and manipulation, after training for 5-18 hours depending on required quality and variations.

Real‐time Neural Rendering of Dynamic Light Fields

“…Our method is unique in the sense that it represents the dynamic light field in an explicit spatio‐temporal cache. We compare it to the two most closely related neural rendering methods, Active Exploration [DPD22] and a NeRF‐style rendering method TiNeuVox [FYW*22]. Both methods are designed with dynamic scenes in mind, making them ideal for comparison.…”

Learning to Learn and Sample BRDFs

“…Adaptivity The sampling pattern is not on‐line [VKŠ*14] or adaptive [DPD22], but the same for each BRDF. A pattern that adapts to some other condition, or maybe to the outcome of previous samples, would be a relevant avenue of future work.…”

“…Bias Moreover, getting the probability of a sample is important in tasks where we want to retain unbiased estimates, such as in Monte Carlo rendering. Note that while we sample unevenly, we do not attempt to divide by the probability density to produce unbiased estimates of gradients, as it is not clear whether a biased gradient estimate can ultimately not be better than an unbiased one [DPD22]. What matters more is that the outer meta-optimizer sees the effect of those gradients and can factor it into the optimization by changing the initialization or step sizes.…”

“…Active or online sampling research [SS18, DPD22] chooses to generate or label the next sample based on metrics inspired by Curriculum Learning, e.g., gradient norm or loss, to allocate more training time to the samples that will reduce the loss the most. The sampling pattern we find transfers across problem instances and does not need to be re‐learned.…”

Neural Free‐Viewpoint Relighting for Glossy Indirect Illumination

“…Additionally, experiments showed that even if this tensor were subsampled at multiple views, the resulting convergence of the network was inadequate to getting good results on novel‐view specularities. In the future, a more adaptive active exploration approach may be helpful to increase the training time spent on hard‐to‐learn examples and prevent overfitting on the diffuse parts of the scene [DPD22].…”