Occluded object detection and exposure in cluttered environments with automated hyperspectral anomaly detection

Abstract

Cluttered environments with partial object occlusions pose significant challenges to robot manipulation. In settings composed of one dominant object type and various undesirable contaminants, occlusions make it difficult to both recognize and isolate undesirable objects. Spatial features alone are not always sufficiently distinct to reliably identify anomalies under multiple layers of clutter, with only a fractional part of the object exposed. We create a multi-modal data representation of cluttered object scenes pairing depth data with a registered hyperspectral data cube. Hyperspectral imaging provides pixel-wise Visible Near-Infrared (VNIR) reflectance spectral curves which are invariant in similar material types. Spectral reflectance data is grounded in the chemical-physical properties of an object, making spectral curves an excellent modality to differentiate inter-class material types. Our approach proposes a new automated method to perform hyperspectral anomaly detection in cluttered workspaces with the goal of improving robot manipulation. We first assume the dominance of a single material class, and coarsely identify the dominant, non-anomalous class. Next these labels are used to train an unsupervised autoencoder to identify anomalous pixels through reconstruction error. To tie our anomaly detection to robot actions, we then apply a set of heuristically-evaluated motion primitives to perturb and further expose local areas containing anomalies. The utility of this approach is demonstrated in numerous cluttered environments including organic and inorganic materials. In each of our four constructed scenarios, our proposed anomaly detection method is able to consistently increase the exposed surface area of anomalies. Our work advances robot perception for cluttered environments by incorporating multi-modal anomaly detection aided by hyperspectral sensing into detecting fractional object presence without need for laboriously curated labels.

Material informed robotics - spectral perception for object identification and parameter inference

“…Sensory gaps in material perception are especially apparent where the deformation of surfaces might cause a failed grasp, or stuck vehicle. In many works, including several not included in the scope of this dissertation [200,201,199], we explored the use of spectroscopy to make an impact in many challenging robot perception problems. We successfully deployed systems that can estimate material class and regress physical quantities in the real world.…”

“…The spectral indices used in this work could still be used as an input to a risk score for the given terrain. This is an extension of the methods proposed in [229], and also builds on our work [201] to conducted automated anomaly detection. We hypothesize the addition of multi-modal, self-supervised approaches will enable finer differentiation between surface terrains.…”