Accurate Passive 3D Polarization Face Reconstruction under Complex Conditions Assisted with Deep Learning

Abstract

Accurate passive 3D face reconstruction is of great importance with various potential applications. Three-dimensional polarization face reconstruction is a promising approach, but one bothered by serious deformations caused by an ambiguous surface normal. In this study, we propose a learning-based method for passive 3D polarization face reconstruction. It first calculates the surface normal of each microfacet at a pixel level based on the polarization of diffusely reflected light on the face, where no auxiliary equipment, including artificial illumination, is required. Then, the CNN-based 3DMM (convolutional neural network; 3D morphable model) generates a rough depth map of the face with the directly captured polarization image. The map works as an extra constraint to correct the ambiguous surface normal obtained from polarization. An accurate surface normal finally allows for an accurate 3D face reconstruction. Experiments in both indoor and outdoor conditions demonstrate that accurate 3D faces can be well-reconstructed. Moreover, with no auxiliary equipment required, the method ensures a total passive 3D face reconstruction.

Data-driven polarimetric imaging: a review

“…The smoothed normal concatenates the fused normal and raw polarization direction images as the input to accurately estimate the surface normal. Subsequently, the skinned multi-person linear (SMPL) representation 182 and deformation stage were used to reconstruct the refined 3D human shape with clothing details rather than naked.…”

Polarization 3D imaging technology: a review

“…Recently, the rapid proliferation of deep learning, which has successful applications in other fields of 3D imaging [69-73], brings the possibility of breaking through the limitations in traditional polarization 3D imaging techniques. An increasing number of researchers have focused on solving the azimuthal ambiguity problem for complex targets in polarimetric 3D imaging through deep learning [51,52,74,75]. Some successful polarization 3D imaging techniques combined with deep learning are outlined below.…”

“…Shao et al [52] proposed a learning-based method for passive 3D polarization face reconstruction. The method uses a CNN-based 3D morphable model (3DMM) to generate a rough depth map of the face from the directly captured polarization image.…”