Photogeometric Scene Flow for High-Detail Dynamic 3D Reconstruction

Abstract

Photometric stereo (PS) is an established technique for high-detail reconstruction of 3D geometry and appearance. To correct for surface integration errors, PS is often combined with multiview stereo (MVS). With dynamic objects, PS reconstruction also faces the problem of computing optical flow (OF) for image alignment under rapid changes in illumination. Current PS methods typically compute optical flow and MVS as independent stages, each one with its own limitations and errors introduced by early regularization. In contrast, scene flow methods estimate geometry and motion, but lack the fine detail from PS. This paper proposes photogeometric scene flow (PGSF) for high-quality dynamic 3D reconstruction. PGSF performs PS, OF, and MVS simulta-neously. It is based on two key observations: (i) while image alignment improves PS, PS allows for surfaces to be relit to improve alignment; (ii) PS provides surface gradients that render the smoothness term in MVS unnecessary, leading to truly data-driven, continuous depth estimates. This synergy is demonstrated in the quality of the resulting RGB appearance, 3D geometry, and 3D motion.

Optimisation of Classic Photometric Stereo by Non-convex Variational Minimisation

“…The ratio procedure described earlier can be used: it simultaneously eliminates the albedo and the nonlinear terms, c.f. [7,21,36,37] and obviously removes the bias due to non-integrability. But let us recall that it is only the best linear unbiased estimate, and also not the optimal one.…”

From Faces to Outdoor Light Probes

“…In addition, the model of [PKA * 09] does not describe face albedo (i.e., a shading-free texture). Thus, we collected a dataset of 13 albedo maps of different subjects using the photometric stereo setup of [GSSM15]. Each albedo map was manually classified and white-balanced to match one of the skintone clusters identified by [WMP * 06].…”

LED-Based Photometric Stereo: Modeling, Calibration and Numerical Solution

“…It could be solved by the characteristic strips expansion method [42,43] if we were dealing with m = 2 images only, but using a larger number of images is necessary in order to design a robust 3D-reconstruction method. Since we are provided with m > 2 images, we follow [20,38,39,41,56,60] and write m 2 PDEs such as (3.22) formed by the m 2 pairs {i, j} ∈ {1, . .…”

“…Lemma 1 If the following condition holds at (ρ * ,z * ): whenever (ρ,z) lies in some small neighborhood of (ρ * ,z * ). 20 And thus a quasi-Newton step with respect to thẽ z-subproblem in (4.5), since ∂Ẽz(z (k) ;ρ (k+1) ,z (k) ) = ∂E ∂z (ρ (k+1) ,z (k) ).…”