A Multiple-View Geometric Model of Specularities on Non-Planar Shapes with Application to Dynamic Retexturing

Abstract

Predicting specularities in images, given the camera pose and scene geometry from SLAM, forms a challenging and open problem. It is nonetheless essential in several applications such as retexturing. A recent geometric model called JOLIMAS partially answers this problem, under the assumptions that the specularities are elliptical and the scene is planar. JOLIMAS models a moving specularity as the image of a fixed 3D quadric. We propose dual JOLIMAS, a new model which raises the planarity assumption. It uses the fact that specularities remain elliptical on convex surfaces and that every surface can be divided in convex parts. The geometry of dual JOLIMAS then uses a 3D quadric per convex surface part and light source, and predicts the specularities by a means of virtual cameras, allowing it to cope with surface's unflatness. We assessed the efficiency and precision of dual JOLIMAS on multiple synthetic and real videos with various objects and lighting conditions. We give results of a retexturing application. Further results are presented as supplementary video material.

A multiple-view geometric model of specularities on non-uniformly curved surfaces

“…We have presented a canonical representation of the previous dual JOLIMAS model [42], where we address the main limitation of the previous approach: the hypothesis of a fixed and constant surface curvature. Our method generalises the JOLIMAS model to any surface geometry while improving the quality of specularity prediction without compromising performance.…”

“…We introduced this 1410 frames sequence in [42] and although the results were already acceptable with an error on average of 2.1%, the central part of the rocket replica does not have a constant curvature, which negatively affected the specularity prediction accuracy of dual JOLIMAS. We observe in figure 10 that canonical dual JOLIMAS outperforms dual JOLIMAS in terms of specularity prediction with an average error of 1.3%.…”

“…In terms of applications, improvements in the specularity prediction capability of canonical dual JOLIMAS will naturally improve 3. http://pointclouds.org the already presented applications such as retexturing [42]. It will also provide a better light source position for local illumination methods to improve the rendering quality of AR applications, since canonical dual JOLIMAS relies on the Torrance-Sparrow local illumination model [54].…”

“…This results in an incorrect 3D ellipsoid reconstruction and incorrect specularity prediction. Figure extracted from[42].…”

“…Figure 2: Illustration of Dual JOLIMAS[42], addressing the issue of specularity prediction on non-planar surfaces. In (a), the computation of virtual cameras Π s,1 , Π s,2 and Π s,3 from the real cameras Π 1 , Π 2 and Π 3 is shown, allowing one to reconstruct an ellipsoid located near the light source.…”