Assistive Grasping Based on Laser-point Detection with Application to Wheelchair-mounted Robotic Arms

Abstract

As the aging of the population becomes more severe, wheelchair-mounted robotic arms (WMRAs) are gaining an increased amount of attention. Laser pointer interactions are an attractive method enabling humans to unambiguously point out objects and pick them up. In addition, they bring about a greater sense of participation in the interaction process as an intuitive interaction mode. However, the issue of human–robot interactions remains to be properly tackled, and traditional laser point interactions still suffer from poor real-time performance and low accuracy amid dynamic backgrounds. In this study, combined with an advanced laser point detection method and an improved pose estimation algorithm, a laser pointer is used to facilitate the interactions between humans and a WMRA in an indoor environment. Assistive grasping using a laser selection consists of two key steps. In the first step, the images captured using an RGB-D camera are pre-processed, and then fed to a convolutional neural network (CNN) to determine the 2D coordinates of the laser point and objects within the image. Meanwhile, the centroid coordinates of the selected object are also obtained using the depth information. In this way, the object to be picked up and its location are determined. The experimental results show that the laser point can be detected with almost 100% accuracy in a complex environment. In the second step, a compound pose-estimation algorithm aiming at a sparse use of multi-view templates is applied, which consists of both coarse- and precise-matching of the target to the template objects, greatly improving the grasping performance. The proposed algorithms were implemented on a Kinova Jaco robotic arm, and the experimental results demonstrate their effectiveness. Compared with commonly accepted methods, the time consumption of the pose generation can be reduced from 5.36 to 4.43 s, and synchronously, the pose estimation error is significantly improved from 21.31% to 3.91%.

Object Affordance-Based Implicit Interaction for Wheelchair-Mounted Robotic Arm Using a Laser Pointer

“…We test the ability of the laser interaction, the intention reasoning model, the learning mechanism, and task execution. Compared with some previous works [41,44], our model achieves some better results.…”

“…Precise detection of the laser spot is crucial for the WMRA to focus on the target objects and subsequently identify their semantics. Instead of using conventional background difference or template matching methods [44], we propose a combined algorithm of YOLOv4 [45] and support vector machine (SVM) to detect laser spots and recognize laser semantics. As shown in Figure 3, the visual system would capture the laser spot and transmit the image stream to the YOLOv4 algorithm.…”

Comprehensive Review on Reaching and Grasping of Objects in Robotics

“…44 Zhong et al have presented a method based on combining laser-point detection and pose estimation algorithms to improve the grasping task with the application of assistive robot arm for wheelchair. 45 In the approach, a three-stage process to achieve pointing and picking up of object has to be carried out. First, a colour-depth (RGB-D) camera is used to capture an image as a pre-processing.…”

Desenvolvimento de um dispositivo para acionamento de joystick de cadeira de rodas motorizada

“…Em (ZHONG et al, 2019) uma caneta laser é utilizada para facilitar a interação com os objetos usando um braço robótico Kinova Jaco ® , ilustrado na Figura 3.…”

“…Figura 3 -Braço robótico montado em Cadeira de Rodas KINOVA JACO ® Fonte: (ZHONG et al, 2019) Em (PALSDOTTIR et al, 2019) uma prova de conceito é realizada para demonstrar o controle de um braço robótico assistivo (também um braço Kinova JACO ® ) por meio de movimento da língua, de forma remota, montado em uma cadeira de rodas.…”