Neural Point Catacaustics for Novel-View Synthesis of Reflections

Abstract

View-dependent effects such as reflections pose a substantial challenge for image-based and neural rendering algorithms. Above all, curved reflectors are particularly hard, as they lead to highly non-linear reflection flows as the camera moves. We introduce a new point-based representation to compute Neural Point Catacaustics allowing novel-view synthesis of scenes with curved reflectors, from a set of casually-captured input photos. At the core of our method is a neural warp field that models catacaustic trajectories of reflections, so complex specular effects can be rendered using efficient point splatting in conjunction with a neural renderer. One of our key contributions is the explicit representation of reflections with a reflection point cloud which is displaced by the neural warp field, and a primary point cloud which is optimized to represent the rest of the scene. After a short manual annotation step, our approach allows interactive high-quality renderings of novel views with accurate reflection flow. Additionally, the explicit representation of reflection flow supports several forms of scene manipulation in captured scenes, such as reflection editing, cloning of specular objects, reflection tracking across views, and comfortable stereo viewing. We provide the source code and other supplemental material on https://repo-sam.inria.fr/fungraph/neural_catacaustics/

3D Gaussian Splatting for Real-Time Radiance Field Rendering

“…A typical neural point-based approach (e.g., [Kopanas et al 2022[Kopanas et al , 2021) computes the color 𝐶 of a pixel by blending N ordered points overlapping the pixel:…”

“…While focusing on specular effects, the diffuse point-based rendering track of Neural Point Catacaustics [Kopanas et al 2022] overcomes this temporal instability by using an MLP, but still required MVS geometry as input. The most recent method [Zhang et al 2022] in this category does not require MVS, and also uses SH for directions; however, it can only handle scenes of one object and needs masks for initialization.…”

HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling

“…Still, due to its impressive performance for static view synthesis, many recent methods build on NeRFs in the quest for higher visual quality, more efficient training, and faster rendering speed [15,53]. Several works improve the quality of NeRFs by accounting for finite pixels and apertures [5,60], by enabling application to unbounded scenes [6,63,64], or by modifying the representation to allow for better reproduction of challenging view-dependent appearances like distorted reflections and refractions [8,17,19,54].…”

ENVIDR: Implicit Differentiable Renderer with Neural Environment Lighting

“…SNISR [44] treats specular highlights as virtual lights underneath the surface with a reflection MLP. Neural Catacaustics [18] uses a neural warp field to approximate the catacaustic through the virtual points for reflections. However, these methods do not model the physically based interaction between lighting and surface, limiting their ability to edit the lighting of represented scenes.…”

“…Prior work largely takes one of two major approaches to address the challenge of learning reflection in neural rendering. The first approach involves explicitly representing virtual lights or images underneath the surface to account for complex view-dependent appearance [14,44,18,36]. The original NeRF [25] and its extensions such as [22,3,47] also synthesize complex reflections in this way (Figure 2).…”