Multimodal Stereo Vision For Reconstruction In The Presence Of Reflection

Abstract

Traditional stereo approaches assume a lambertian scene, an assumption which is violated in the presence of specular reflections. A variety of techniques have been developed to detect and reconstruct these surfaces [1, 4] using a variety of constraints, however in this work we attempt to reconstruct a reflecting surface and a reflected scene using different imaging modalities. Using a four camera system shown in Fig. 1, operating as two stereo pairs we reconstruct a reflecting surface as well as a reflected scene. The camera system consists of a pair of visible band cameras and a pair of Long Wave InfraRed (LWIR) cameras. We leverage these different imaging modalities by taking advantage of the fact that reflectivity is wavelength dependent and reflective materials in one modality can appear non reflective in complementing modalities.We calibrate the system by extending the work of [3] to handle cross modality calibration. This gives us a common coordinate system. To reconstruct a reflected scene we first extract the reflecting surface. This is done by adding texture to the surface in only one modality, and reconstructing correspondences, and fitting a plane to these points using principle component analysis. The implicit representation of this planecan be used to intersect rays for reconstruction. Where P is the origin, n is the plane normal and d is a constant. Reconstructing the scene is accomplished by stereo matching the reflected images using uncalibrated rectification and disparity matching using [2]. Each point in the disparity map can be viewed as a ray going from the camera center through the image plane defined by,where C 0 is the camera center, andwhere A is the camera matrix, and R is the camera rotation matrix, and [x i , y i ] is the pixel location. We intersect these rays with the reflecting surface by substituting P from equation 1 with V i from equation 2, and the reflected ray direction is defined as,

LAT: Local area transform for cross modal correspondence matching

“…In real world, however, images are acquired in an uncontrolled environment; thus, the image may suffer intensity deformations due to changes in illumination conditions, camera photometric parameters, viewing positions, and so on [16]. Furthermore, recently, cross-modality imaging system (e.g., multi-spectral imaging system [17,18] has been attracted many attentions to address challenging problems occurring in the conventional unimodal imaging system. Images acquired from different modalities also have intensity deformations due to changes in sensor responses and spectral distributions.…”