Variational Shape and Reflectance Estimation Under Changing Light and Viewpoints

Abstract

Abstract. Fitting parameterized 3D shape and general reflectance models to 2D image data is challenging due to the high dimensionality of the problem. The proposed method combines the capabilities of classical and photometric stereo, allowing for accurate reconstruction of both textured and non-textured surfaces. In particular, we present a variational method implemented as a PDE-driven surface evolution interleaved with reflectance estimation. The surface is represented on an adaptive mesh allowing topological change. To provide the input data, we have designed a capture setup that simultaneously acquires both viewpoint and light variation while minimizing self-shadowing. Our capture method is feasible for real-world application as it requires a moderate amount of input data and processing time. In experiments, models of people and everyday objects were captured from a few dozen images taken with a consumer digital camera. The capture process recovers a photo-consistent model of spatially varying Lambertian and specular reflectance and a highly accurate geometry.

Generic Scene Recovery Using Multiple Images

“…Note that, unlike [15,16], we do not use any thresholding to filter out specular highlight pixels. The result for the smoothed "bimba" data set is shown in Fig.…”

“…As shown, the method of [2] fails to get a good shape even when the shape is very simple, while our method estimates it accurately. Also, with such images, given the large proportion of overbright surface parts, it seems intuitive that the strategy chosen by [16] and [15] (who consider bright pixels as outliers) might return less accurate results, because it removes too much information.…”

“…So, the method in [13] is a "multiview shape from shading" method, similarly as the one proposed by [8,14] which focus on the Lambertian case. To our knowledge, with the exception of [15,16], all the works going in the same direction as ours are limited to surfaces made of a single (textureless) material. In particular, this is the case for the photometric stereo methods proposed by [17,18] and for the multiview photometric stereo work of [19].…”

“…In particular, this is the case for the photometric stereo methods proposed by [17,18] and for the multiview photometric stereo work of [19]. Only the similar works [15,16] are able to recover scenes with varying albedo. However, in [15,16], the authors tried to filter out specular highlights by using a simple thresholding and to use only diffuse components to estimate the shape.…”

“…Only the similar works [15,16] are able to recover scenes with varying albedo. However, in [15,16], the authors tried to filter out specular highlights by using a simple thresholding and to use only diffuse components to estimate the shape. [15] also used a thresholding to detect shadowed pixels that are not visible from light sources, which is however not working under multiple light sources.…”