Egocentric Gesture Recognition Using 3D Convolutional Neural Networks for the Spatiotemporal Adaptation of Collaborative Robots

Abstract

Collaborative robots are currently deployed in professional environments, in collaboration with professional human operators, helping to strike the right balance between mechanization and manual intervention in manufacturing processes required by Industry 4.0. In this paper, the contribution of gesture recognition and pose estimation to the smooth introduction of cobots into an industrial assembly line is described, with a view to performing actions in parallel with the human operators and enabling interaction between them. The proposed active vision system uses two RGB-D cameras that record different points of view of gestures and poses of the operator, to build an external perception layer for the robot that facilitates spatiotemporal adaptation, in accordance with the human's behavior. The use-case of this work is concerned with LCD TV assembly of an appliance manufacturer, comprising of two parts. The first part of the above-mentioned operation is assigned to a robot, strengthening the assembly line. The second part is assigned to a human operator. Gesture recognition, pose estimation, physical interaction, and sonic notification, create a multimodal human-robot interaction system. Five experiments are performed, to test if gesture recognition and pose estimation can reduce the cycle time and range of motion of the operator, respectively. Physical interaction is achieved using the force sensor of the cobot. Pose estimation through a skeleton-tracking algorithm provides the cobot with human pose information and makes it spatially adjustable. Sonic notification is added for the case of unexpected incidents. A real-time gesture recognition module is implemented through a Deep Learning architecture consisting of Convolutional layers, trained in an egocentric view and reducing the cycle time of the routine by almost 20%. This constitutes an added value in this work, as it affords the potential of recognizing gestures independently of the anthropometric characteristics and the background. Common metrics derived from the literature are used for the evaluation of the proposed system. The percentage of spatial adaptation of the cobot is proposed as a new KPI for a collaborative system and the opinion of the human operator is measured through a questionnaire that concerns the various affective states of the operator during the collaboration.

Computational ergonomics for task delegation in Human-Robot Collaboration: spatiotemporal adaptation of the robot to the human through contactless gesture recognition

“…The objects involved in this TV assembly process are a TV frame and two circuit cards: the power supply (PSUgold card) and the mainboard (chassis-green card). An earlier study [24] demonstrated the network's ability to recognize the 11 gestures with a 98.5% accuracy.…”

“…To summarize, highly adaptable and rapidly reconfigurable frameworks with real-time data processing capabilities are necessary to handle the industry's ergonomics challenges while simultaneously ensuring productivity. This paper expands on the authors' prior work on adaptive HRC [24], and automatic ergonomic monitoring [25], [26], [27], with the goal of creating a unified ergonomic and reconfigurable HRC framework.…”

“…To facilitate natural collaboration and assist the operator in performing only ergonomically safe motions, a posture estimation module was added that enables the robot to spatially adapt to the operator. Papanagiotou et al [24] presented this posture estimation module and spatial adaptation procedure in a previous study. The spatial adaptation refers to the fact that the robotic arm does not place the cards in a fixed position but rather adapts to the operator's anthropometric characteristics.…”

“…As a key performance indicator (KPI), the percentage of robot spatial adaptation was used [24]. In addition, this paper presents a second KPI for determining how much operator effort is reduced as a result of the gesture recognition module.…”

An Outlook into the Future of Egocentric Vision

“…In addition to human-computer interaction, the concept has also been extended to human-robot interaction. Papanagiotou et al (2021) proposed a multi-task approach including gesture recognition to enable human-robot collaboration on an industrial assembly line. The main component is represented by a gesture recognition module which is based on 3D CNN trained on egocentric data acquired with a GoPro camera.…”