In-Hand Pose Estimation Using Hand-Mounted RGB Cameras and Visuotactile Sensors

Abstract

This paper proposes a method to estimate the 6D pose of an object grasped by a robot hand using RGB cameras mounted on the palm and visuotactile sensors installed at the fingertips. It can deal with objects made from a wide range of materials thanks to combining the two types of sensors. The method allows a robot to robot to perform in-hand pose estimation while holding the object, eliminating the need for preparatory actions or particular environmental backgrounds. The mechanism at the back of the method includes deep-learning-based background subtraction and denoising auto-encoder-based sensor fusion. With the poses estimated using the proposed method, a robot controller can rectify the grasping uncertainty and adjust the robot motion to move an object toward required goals with satisfying accuracy. We conduct various studies and analyses in the experimental section to understand the proposed method's advantages and disadvantages. The results demonstrate the benefits of the proposed combination and mechanism. They also provide essential knowledge to readers considering using a similar configuration for estimating object poses.

Bio-Inspired Proprioceptive Touch of a Soft Finger with Inner-Finger Kinesthetic Perception

“…Despite lacking a standard experiment setup for in-hand object pose estimation, some work has still been explored. The authors of [33] used RGB cameras and GelSights to estimate in-hand object pose with 15 mm accuracy in position and 15 degrees accuracy in orientation. The authors of [34] fused vision and tactile data to obtain 2.99 mm accuracy in position and 8.074 degrees accuracy in orientation.…”