3D Shape Perception from Monocular Vision, Touch, and Shape Priors

Abstract

Perceiving accurate 3D object shape is important for robots to interact with the physical world. Current research along this direction has been primarily relying on visual observations. Vision, however useful, has inherent limitations due to occlusions and the 2D-3D ambiguities, especially for perception with a monocular camera. In contrast, touch gets precise local shape information, though its efficiency for reconstructing the entire shape could be low. In this paper, we propose a novel paradigm that efficiently perceives accurate 3D object shape by incorporating visual and tactile observations, as well as prior knowledge of common object shapes learned from large-scale shape repositories. We use vision first, applying neural networks with learned shape priors to predict an object's 3D shape from a single-view color image. We then use tactile sensing to refine the shape; the robot actively touches the object regions where the visual prediction has high uncertainty. Our method efficiently builds the 3D shape of common objects from a color image and a small number of tactile explorations (around 10). Our setup is easy to apply and has potentials to help robots better perform grasping or manipulation tasks on real-world objects.

Mental geometry of 3D size and shape perception

“…Ames Chair (Ittelson, 1952), would be seen as unbroken and cohesive from the proper viewpoint, without having to invoke stronger assumptions of simplicity or regularity in reconstructing the 3D world (Li & Pizlo, 2011). The "generic viewpoint assumption" (Freeman, 1994) thus implicitly contains a "generic projective geometry assumption", and this can be made more concrete in object and scene perception by incorporating priors that assume the invariants of projective geometry, especially when extracting 3D shapes from contours (Li, Pizlo & Steinman, 2009;Elder, 2018;Wang et al, 2018). The fact that slant is ambiguous in perspective projection is compatible with some real-world illusions of slant and non-rigidity (Griffiths & Zaidi, 1998, 2000.…”

Exploiting Visual-Outer Shape for Tactile-Inner Shape Estimation of Objects Covered with Soft Materials

“…Integrating visual and tactile information for object shape estimation [13]- [16] as well as object recognition [28] has been studied. However, these studies commonly assumed that the visual and tactile shapes are very similar to the actual object shape.…”

“…To improve the efficiency of tactile exploration, exploiting visual information obtained through cameras has been explored since visual information can be in a wide range without physical touch. For example, both visual and tactile information is simply used in a mixed way [13], [14], or utilizing uncertainty of shapes constructed from visual information for active tactile shape estimation [15], [16]. However, it is not straightforward for objects covered with soft materials, since the visual information captures the outer shape of the covered materials, and it is unknown how much such visual-outer shape is similar/dissimilar to the tactile-inner shape.…”

Multi-Modal Geometric Learning for Grasping and Manipulation

“…2. This work is differentiated from others [1] in that our CNN is acting on both the depth and tactile as input information fed directly into the model rather than using the tactile information to update the output of a CNN not explicitly trained on tactile information. This enables the tactile information to produce non-local changes in the resulting mesh.…”