A Deep Learning-Based Hand-eye Calibration Approach using a Single Reference Point on a Robot Manipulator

Abstract

We present a hand-eye calibration approach based on a deep learning-based regression architecture to find the transformation between the robot end-effector and an external camera. For this, we hypothesise that it is possible to track a single reference point in the robot's end-effector to estimate the hand-eye geometric transformation using a deep neural network and a 3D vision system. To explore this hypothesis, we design three experiments to study the different components of our proposed network architecture while solving isolated cases of the hand-eye calibration problem. Our experimental results using a simulated environment show that our proposed approach has less than 1 mm error for translation and less than 2.31 degrees error for orientation. We also carried out experiments for our third approach in two real robotic testbeds (a Universal Robot 3 and the Rethink Baxter robot). Our approach achieves 2 mm and 5.9 degrees, 4.53 mm and 9.2 degrees of errors for the Universal Robot UR3 and the Rethink Baxter robot.

Continual Learning Approaches to Hand-Eye Calibration in Robots

“…We considered the hand-eye calibration problem as a regression problem using deep learning. In our previous work, we developed an endto-end deep learning-based hand-eye calibration approach [5] and conducted experiments in a simulated environment and two real robotic environments (the Rethink Baxter and a Universal Robot 3). In this study, we tailored this approach to extend learning hand-eye calibration space via Continual Learning.…”

“…Valassakis et al [6] employed an end-to-end deep learning architecture that directly estimates the hand-eye calibration parameters, where the camera is attached to the end-effector of the robot. In our previous work [5], we proposed a deep-learning architecture that estimates the hand-eye calibration parameters directly from a single pair of RGB and depth images. Although [6] and [5] achieved flexibility for a wide range of hand-eye calibration spaces, they are trained offline and cannot adapt to changes in data distribution and the robotic environment on-the-fly.…”

“…Bahadir et al [5] utilises supervised learning to estimate the camera's pose from RGB and depth images and the pose of a single reference point on the robot manipulator. To achieve this, [5] employs two separate encoders (depicted in Figure 1) to extract features from the RGB and depth images, which are then concatenated with the reference point's pose. The resulting feature vectors are fed into a neural network that estimates either the translation or orientation component of the camera pose.…”

“…The resulting feature vectors are fed into a neural network that estimates either the translation or orientation component of the camera pose. By adopting this approach, [5] avoids the need for a calibration target, as in traditional approaches such as Tsai's hand-eye calibration [1], and implicitly handles the camera extrinsic parameters for each end-effector's pose.…”

“…Recent advancements in deep learning architectures, such as convolutional neural networks, have led to the development of hand-eye calibration systems [5][6][7] capable of estimating different camera poses with respect to a single robot base, thus providing greater flexibility and adaptability to industrial manufacturing systems. However, these deep learning-based approaches have limitations.…”