Refractive Light-Field Features for Curved Transparent Objects in Structure From Motion

Abstract

Curved refractive objects are common in the human environment, and have a complex visual appearance that can cause robotic vision algorithms to fail. Light-field cameras allow us to address this challenge by capturing the view-dependent appearance of such objects in a single exposure. We propose a novel image feature for light fields that detects and describes the patterns of light refracted through curved transparent objects. We derive characteristic points based on these features allowing them to be used in place of conventional 2D features. Using our features, we demonstrate improved structure-from-motion performance in challenging scenes containing refractive objects, including quantitative evaluations that show improved camera pose estimates and 3D reconstructions. Additionally, our methods converge 15-35% more frequently than the state-of-the-art. Our method is a critical step towards allowing robots to operate around refractive objects, with applications in manufacturing, quality assurance, pick-and-place, and domestic robots working with acrylic, glass and other transparent materials.

3d Digitization of Transparent and Glass Surfaces: State of the Art and Analysis of Some Methods

“…Some researchers (Morris and Kutulakos, 2011;Ding et al, 2011) employed stereo/multiple cameras to record the refractive surface, relying on a cross-view normal consistency constraint: the normals computed using the pixel-point correspondences obtained from multiple viewpoints must be consistent. Alternatively, some studies have been conducted (Ye et al, 2012;Wetzstein et al, 2011;Tsai, 2020;Tsai et al, 2021) to estimate ray-ray correspondences utilizing specific devices such as Bokode (Ye et al, 2012) and light field probes (Wetzstein et al, 2011;Tsai et al, 2021) by capturing the incident rays released from the background and the exiting rays traveling to the camera. Although 3D results appear to be highly promising, the high cost of such devices is an important downside of these approaches.…”