Object Localization and Depth Estimation for Eye-in-Hand Manipulator Using Mono Camera

Abstract

This paper proposes the object localization and depth estimation to select and set goals for robots via machine vision. An algorithm based on a deep region-based convolution neural network (R-CNN) will recognize targets and non-targets. After the targets are recognized, we employed both the k-nearest neighbors (kNN) and the fuzzy inference system (FIS) to localize two-dimension (2D) positions. Moreover, based on the field of view (FoV) and a disparity map, the depth is estimated by a mono camera mounted on the end-effector with an eye-in-hand manipulator structure. Although using a single mono camera, the system can easily find the camera baseline by only shifting the end-effector a few millimeters towards the x-axis. Thus, we can obtain and identify the depth of the layered environment in 3D points, which form a dataset to recognize the junction box covers on the table. Experimental tests confirmed that the algorithm could accurately distinguish junction box covers or non-targets and could estimate whether the targets are within the depth for grasping by three-finger grippers. Furthermore, the proposed optimized depth error of-0.0005%, and localization method could precisely position the junction box cover with recognizing and picking error rates 0.993 and 98.529% respectively. INDEX TERMS Region-based convolution neural network, eye-in-hand manipulator, machine vision, robotics, and automation.

An Artificial Intelligence of Things-Based Picking Algorithm for Online Shop in the Society 5.0’s Context

“…The CNN, RCNN, Fast RCNN, and Faster RCNN are trained to perform bounding box classification and regression simultaneously [ 23 ]. In practice, YOLO runs much faster than Faster RCNN due to its simpler architecture.…”

“…In the traditional Speeded Up Robust Features (SURF) algorithm, the box filter is used to approximate the convolution of the 2D images [ 40 , 41 ]. The Gaussian second- order derivative can simplify the calculation and improve the efficiency [ 23 ]. The SURF detector uses Hessian matrix because of its good performance in computation time and accuracy.…”

“…Several techniques to achieve this goal use a mono camera, stereo camera, or depth camera. An RGB-D camera is prevalent among them, but its construction is still a little bigger than a mono camera [ 23 ]. Especially, it will be an obstacle when entering narrow gaps if the eye-in-hand mode is used.…”

“…In our prior work [ 23 ], object localization and depth estimation had implemented the R-CNN on eye-in-hand construction using a mono camera, which recognized and sorted the target and then estimated the target’s depth ( Z ) and centroid. At the same time, the other pair coordinates ( XY ) are obtained from the left camera and right camera disparity.…”

Motion Control Analysis of Two Collaborative Arm Robots in Fruit Packaging System

“…Image processing is a key point for the visual cue of a robot used in a place where it is necessary to recognize an object, such as an agricultural robot [8,9,10,11,12,13,14,15,16,17]. Image processing is helpful in any agricultural situation, such as seeding [8][9], maintenance [18] [19], yield forecast [20][21], harvesting [12], and packaging [17,22,23,24,25,26,27]. The use of robots in agriculture could ease the farmer's work burden and increase the harvest yield [8].…”