Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo

Abstract

We present a modern solution to the multi-view photometric stereo problem (MVPS). Our work suitably exploits the image formation model in a MVPS experimental setup to recover the dense 3D reconstruction of an object from images. We procure the surface orientation using a photometric stereo (PS) image formation model and blend it with a multi-view neural radiance field representation to recover the object's surface geometry. Contrary to the previous multi-staged framework to MVPS, where the position, isodepth contours, or orientation measurements are estimated independently and then fused later, our method is simple to implement and realize. Our method performs neural rendering of multi-view images while utilizing surface normals estimated by a deep photometric stereo network. We render the MVPS images by considering the object's surface normals for each 3D sample point along the viewing direction rather than explicitly using the density gradient in the volume space via 3D occupancy information. We optimize the proposed neural radiance field representation for the MVPS setup efficiently using a fully connected deep network to recover the 3D geometry of an object. Extensive evaluation on the DiLiGenT-MV benchmark dataset shows that our method performs better than the approaches that perform only PS or only multi-view stereo (MVS) and provides comparable results against the state-of-the-art multistage fusion methods.

A Digital 4D Information System on the World Scale: Research Challenges, Approaches, and Preliminary Results

“…Another approach bypasses the modelling stage to generate visualizations directly from imagery [127,133,134], e.g., by transforming or assembling image content (recent image generators like DALL-E [135]). Recent approaches include neural radiance fields (NeRFs) [136,137], which predict shifting spatial perspectives even from single images [138], and can predict 3D geometries [139]. Generative adversarial networks (GANs) are a combination of proposal and evaluation components of machine learning.…”

GeoNeRF: Generalizing NeRF with Geometry Priors

“…DS-NeRF [11] shows that depth supervision can help NeRF train faster with fewer input views. Moreover, numerous works [24,38,52,58] show that despite the highquality color rendering, NeRF has difficulty reconstructing 3D geometry and surface normals. Accordingly, for training samples coming from datasets with ground truth depths, we also output the predicted depth d for each ray and supervise it if the ground truth depth of that pixel is available:…”