Distinguishing Refracted Features Using Light Field Cameras With Application to Structure From Motion

Abstract

Robots must reliably interact with refractive objects in many applications; however, refractive objects can cause many robotic vision algorithms to become unreliable or even fail, particularly feature-based matching applications, such as structure-from-motion. We propose a method to distinguish between refracted and Lambertian image features using a light field camera. Specifically, we propose to use textural cross-correlation to characterise apparent feature motion in a single light field, and compare this motion to its Lambertian equivalent based on 4D light field geometry. Our refracted feature distinguisher has a 34.3% higher rate of detection compared to state-of-the-art for light fields captured with large baselines relative to the refractive object. Our method also applies to light field cameras with much smaller baselines than previously considered, yielding up to 2 times better detection for 2D-refractive objects, such as a sphere, and up to 8 times better for 1D-refractive objects, such as a cylinder. For structure from motion, we demonstrate that rejecting refracted features using our distinguisher yields up to 42.4% lower reprojection error, and lower failure rate when the robot is approaching refractive objects. Our method lead to more robust robot vision in the presence of refractive objects.

Refractive Light-Field Features for Curved Transparent Objects in Structure From Motion

“…Tsai et al extended this work to show that a 3D point manifests as a plane in 4D that has two orthogonal normal vectors, which yielded more accurate estimates of how closely an image feature follows the Lambertian model. These estimates helped distinguish more types of refractive objects with a higher rate of detection in order to reject refracted scene content [2].…”

“…In this paper, we propose a novel RLFF based on the appearance of background texture through a refractive object. We extend [2] to derive novel methods for detecting, extracting and estimating the 4D structure of an RLFF in the LF. We use the full LF to detect and extract each feature, making maximal use of available information.…”

“…We employ the proposed RLFF to allow SfM to operate in scenes dominated by refractive objects. Notably, while prior work in [2] focused on detecting and rejecting the refracted scene content, our approach directly uses both the Lambertian and refracted scene content for more reliable camera pose and 3D shape estimation.…”

“…Previous work has shown LF capture offers advantages in detecting reliable features [7] and ignoring refracted features [2]. None has to our knowledge described the detection and use of refracted features for robotic applications.…”

“…Their view-dependent nature violates assumptions that underpin conventional vision algorithms, which can prevent them from operating as intended [1], [2]. LF cameras capture dense and uniformly-sampled multiple views of the scene in one exposure [2], [6]. A single LF image describes view-dependent effects such as occlusion, specular reflection, and notably, refraction.…”