Integration of Industrially-Oriented Human-Robot Speech Communication and Vision-Based Object Recognition

Abstract

This paper presents a novel method for integration of industrially-oriented human-robot speech communication and vision-based object recognition. Such integration is necessary to provide context for task-oriented voice commands. Context-based speech communication is easier, the commands are shorter, hence their recognition rate is higher. In recent years, significant research was devoted to integration of speech and gesture recognition. However, little attention was paid to vision-based identification of objects in industrial environment (like workpieces or tools) represented by general terms used in voice commands. There are no reports on any methods facilitating the abovementioned integration. Image and speech recognition systems usually operate on different data structures, describing reality on different levels of abstraction, hence development of context-based voice control systems is a laborious and time-consuming task. The aim of our research was to solve this problem. The core of our method is extension of Voice Command Description (VCD) format describing syntax and semantics of task-oriented commands, as well as its integration with Flexible Editable Contour Templates (FECT) used for classification of contours derived from image recognition systems. To the best of our knowledge, it is the first solution that facilitates development of customized vision-based voice control applications for industrial robots.

Improved Multi-Stream Convolutional Block Attention Module for sEMG-Based Gesture Recognition

“…Attentional mechanisms ( Hao et al, 2021b ) has aroused the interest of many researchers because it have fewer parameters, faster speed, and better results in important areas such as machine translation ( Hao et al, 2021a ), speech recognition ( Bahdanau et al, 2016 ; Rogowski et al, 2020 ), image recognition ( Wang et al, 2017 ), and gradually started to be applied in sEMG gesture recognition ( Hu J. et al, 2018 ; Jiang et al, 2019c ; Liu et al, 2021 ) Previous CNN gesture recognition models often do not give enough attention to the characteristics of the EMG signal and do not make full use of the temporal information ( Atzori et al, 2014b ; Atzori et al, 2016 ; Geng et al, 2016 ; Ketykó et al, 2019 ). Therefore, this paper introduces a convolutional attention mechanism into the EMG gesture recognition method and designs a one-dimensional convolutional attention module based on time and feature channels to make it more applicable to EMG gesture recognition.…”

Semantic Trajectory Planning for Industrial Robotics

“…By adeptly detecting and recognizing objects within the robot's working domain, the potential for collisions and accidents is effectively minimized. A notable study delves into the synthesis of vision-based object recognition and human-robot communication, shedding light on the profound importance of this synergy within industrial contexts (Rogowski et al, 2020).…”

Multi-Objective Location and Mapping Based on Deep Learning and Visual Slam

“…This way of building maps can construct a more expressive map of the environment to assist the robot to better understand the environment as a whole, but it is not good for the robot to recognize the individuals in the environment. Robots cannot interact with individuals in the environment because they cannot distinguish between different individuals in the environment, which limits the application of intelligent robots to a certain extent [ 30 , 31 , 32 ]. The main contributions of this paper are as follows: Propose a method of target tracking combined with instance segmentation, which can segment the objects in an image and track each object to establish data associations for different frames.…”