ShapeMap 3-D: Efficient shape mapping through dense touch and vision

Suresh, Sudharshan; Si, Zilin; Mangelson, Joshua G.; Yuan, Wenzhen; Kaess, Michael

doi:10.1109/icra46639.2022.9812040

Cited by 34 publications

(15 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Combining different sensor data is beneficial compared to a single one. Suresh et al [42] employed an overlooking depth camera to initialize observation and incrementally built an object's 3D shape through multiple touches. Anzai et al [43] used a hand-mounted RGB camera and a GelSight sensor attached to the robot hand to estimate changes in the grasping posture from an initial position.…”

Section: Methods Using Multiple Sensorsmentioning

confidence: 99%

In-Hand Pose Estimation Using Hand-Mounted RGB Cameras and Visuotactile Sensors

et al. 2023

View full text Add to dashboard Cite

This paper proposes a method to estimate the 6D pose of an object grasped by a robot hand using RGB cameras mounted on the palm and visuotactile sensors installed at the fingertips. It can deal with objects made from a wide range of materials thanks to combining the two types of sensors. The method allows a robot to robot to perform in-hand pose estimation while holding the object, eliminating the need for preparatory actions or particular environmental backgrounds. The mechanism at the back of the method includes deep-learning-based background subtraction and denoising auto-encoder-based sensor fusion. With the poses estimated using the proposed method, a robot controller can rectify the grasping uncertainty and adjust the robot motion to move an object toward required goals with satisfying accuracy. We conduct various studies and analyses in the experimental section to understand the proposed method's advantages and disadvantages. The results demonstrate the benefits of the proposed combination and mechanism. They also provide essential knowledge to readers considering using a similar configuration for estimating object poses.

show abstract

Section: Methods Using Multiple Sensorsmentioning

confidence: 99%

In-Hand Pose Estimation Using Hand-Mounted RGB Cameras and Visuotactile Sensors

et al. 2023

View full text Add to dashboard Cite

show abstract

“…However, these are not as effective as an uncertainty-driven approach. Uncertainty can come from the Gaussian distribution [16]- [19], [21]; from the Monte Carlo dropout [24]; or from the Signed Distance Function (SDF) [1], [25]. Alternatively, it can be learned where to touch as in Smith et al [23].…”

Section: Visuo-haptic Shape Completionmentioning

confidence: 99%

Efficient Visuo-Haptic Object Shape Completion for Robot Manipulation

Rustler¹,

Matas²,

Hoffmann³

2023

Preprint

View full text Add to dashboard Cite

For robot manipulation, a complete and accurate object shape is desirable. Here, we present a method that combines visual and haptic reconstruction in a closed-loop pipeline. From an initial viewpoint, the object shape is reconstructed using an implicit surface deep neural network. The location with highest uncertainty is selected for haptic exploration, the object is touched, the new information from touch and a new point cloud from the camera are added, object position is re-estimated and the cycle is repeated. We extend Rustler et al. (2022) by using a new theoretically grounded method to determine the points with highest uncertainty, and we increase the yield of every haptic exploration by adding not only the contact points to the point cloud but also incorporating the empty space established through the robot movement to the object. Additionally, the solution is compact in that the jaws of a closed two-finger gripper are directly used for exploration. The object position is re-estimated after every robot action and multiple objects can be present simultaneously on the table. We achieve a steady improvement with every touch using three different metrics and demonstrate the utility of the better shape reconstruction in grasping experiments on the real robot. On average, grasp success rate increases from 63.3% to 70.4% after a single exploratory touch and to 82.7% after five touches. The collected data and code are publicly available (https://osf. io/j6rkd/, https://github.com/ctu-vras/vishac).

show abstract

“…GelSight sensor [2] and various revised ones [4,8,9,11] use the photometric stereo technique [3] to measure highresolution 3D geometry of contact objects with a monocular camera. With a GelSight sensor mounted on the gripper, robots can accomplish challenging tasks such as texture recognition [12,13,14], dexterous manipulation [15,16,17,18], shape mapping [19,20], as well as liquid property estimation [21].…”

Section: Related Workmentioning

confidence: 99%

DTact: A Vision-Based Tactile Sensor that Measures High-Resolution 3D Geometry Directly from Darkness

Lin¹,

Lin²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Vision-based tactile sensors that can measure 3D geometry of the contacting objects are crucial for robots to perform dexterous manipulation tasks. However, the existing sensors are usually complicated to fabricate and delicate to extend. In this work, we novelly take advantage of the reflection property of semitransparent elastomer to design a robust, low-cost, and easy-to-fabricate tactile sensor named DTact. DTact measures high-resolution 3D geometry accurately from the darkness shown in the captured tactile images with only a single image for calibration. In contrast to previous sensors, DTact is robust under various illumination conditions. Then, we build prototypes of DTact that have non-planar contact surfaces with minimal extra efforts and costs. Finally, we perform two intelligent robotic tasks including pose estimation and object recognition using DTact, in which DTact shows large potential in applications.

show abstract

ShapeMap 3-D: Efficient shape mapping through dense touch and vision

Cited by 34 publications

References 43 publications

In-Hand Pose Estimation Using Hand-Mounted RGB Cameras and Visuotactile Sensors

In-Hand Pose Estimation Using Hand-Mounted RGB Cameras and Visuotactile Sensors

Efficient Visuo-Haptic Object Shape Completion for Robot Manipulation

DTact: A Vision-Based Tactile Sensor that Measures High-Resolution 3D Geometry Directly from Darkness

Contact Info

Product

Resources

About