Action recognition algorithm from visual sensor data for contactless robot control systems

Voronin, Viacheslav; Zhdanova, Marina; Khamidullin, Ilia; Semenishchev, Evgeny A.; Zelensky, Alexander A.; Ilyukhin, Yu.

doi:10.1117/12.2642003

Cited by 3 publications

(9 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The experiment showed that remote gesture control systems could control the robot. The presented improved algorithm showed an almost 2% higher efficiency than in previous studies [19].…”

Section: Discussionmentioning

confidence: 57%

See 1 more Smart Citation

Action recognition algorithm from multimodal images for human-robot collaboration systems

Voronin¹,

Zhdanova²,

Semenishchev³

et al. 2023

Multimodal Sensing and Artificial Intelligence: Technologies and Applications III

View full text Add to dashboard Cite

Automation of production processes using robots is a priority for developing many industrial enterprises. Human-machine interaction is a key component of such control infrastructure. The proposed algorithm is a four-stage procedure: (a) fusion information from multimodal sensors based on the quaternion model, (b) image preprocessing using a 3D Gabor filter, (c) a descriptor calculation using 3D local binary dense micro-block difference with skeleton points, and (d) classification. The proposed algorithm is based on capturing 3D sub-volumes located inside a video sequence patch and calculating the difference in intensities between these sub-volumes; for intensified motion, used the convolution with a bank of 3D arbitrarily oriented Gabor filters and calculating 3D local binary dense micro-block difference. A program was developed for transmitting information about the target points of the robot's movement through a virtual TCP / IP port and a script for working out the target points in the simulation environment. To test the effectiveness of the proposed algorithm, we simulate an action recognition system for Human-robot collaboration in the RoboGuide environment.

show abstract

“…The experiment showed that remote gesture control systems could control the robot. The presented improved algorithm showed an almost 2% higher efficiency than in previous studies [19].…”

Section: Discussionmentioning

confidence: 57%

“…Each action was performed 4 times by 15 people. Thus, 480 videos of gesture commands were obtained [19].…”

Section: Introductionmentioning

confidence: 99%

Action recognition algorithm from multimodal images for human-robot collaboration systems

Voronin¹,

Zhdanova²,

Semenishchev³

et al. 2023

Multimodal Sensing and Artificial Intelligence: Technologies and Applications III

View full text Add to dashboard Cite

show abstract

“…Next, we use an approach based on the classical Laplace pyramid [1] described in Section 2.1 to fuse data. The fused data is preprocessed in the next step by a two-way Gabor Quaternion Fourier Transform (GQFT) in [2,3,4] and calculating 3D local binary dense micro-block difference. Also, we obtain human skeleton data using a convolutional neural network or some motion capture devices (for example, Kinect) [4].…”

Section: Proposed Methodsmentioning

confidence: 99%

“…The fused data is preprocessed in the next step by a two-way Gabor Quaternion Fourier Transform (GQFT) in [2,3,4] and calculating 3D local binary dense micro-block difference. Also, we obtain human skeleton data using a convolutional neural network or some motion capture devices (for example, Kinect) [4]. Next, the 3D local binary dense microblock difference descriptor and the skeletal descriptor are concatenated into a single feature vector.…”

Section: Proposed Methodsmentioning

confidence: 99%

A multimodal visual guided robot collaborative system based on the classification of multi-class human motion

Voronin,

Zhdanova,

Tokareva

et al. 2023

Advanced Optical Imaging Technologies VI

View full text Add to dashboard Cite

The key technology of modern intelligent production is the concept of human-robot interaction. Operators and robots collaborate to perform complex tasks in various scenarios in such a system. Understanding human behavior introduces challenges for collaborative robot systems to perform efficient tasks in unstructured and dynamic environments. The proposed algorithm is based on classifying multi-class human motion from multimodal visual data. For this purpose, we apply the Laplace pyramid for visible, depth, and thermal data from imaging sensors. To prevent the appearance of visual artifacts, a multiscale approach is proposed for combining images based on the Laplace pyramid and calculating the optimal weights. The next stage is fused data preprocessing by the two-sided Gabor quaternionic Fourier transform and calculating 3D local binary dense micro-block difference. Also, we obtain human skeleton data using a convolutional neural network. Based on the coordinates of the unique points of the human body skeleton and the distances between them, a descriptor is constructed that describes the person's posture on each frame and considers the time information between frames. At the next stage, the 3D local binary dense micro-block difference and skeletal descriptor are combined into a single feature vector. Moreover, finally, this descriptor is classified by a neural network. We present simulation modeling on the effectiveness of the proposed action recognition algorithm in the RoboGuid environment.

show abstract

“…Creating an interface that is understandable and ergonomic for a person, in this vein, becomes an almost impossible task. One solution to this problem can be to control the device using a command of gestures given by a person [1][2][3][4]. This artificial cognitive "sensory perception" or ability is a communication channel between humans and robots.…”

Section: Introductionmentioning

confidence: 99%

Real-time robotic hand control using human gesture recognition

Egipko

Voronin

Gapon

et al. 2023

Real-Time Image Processing and Deep Learning 2023

View full text Add to dashboard Cite

Currently, there are many options for controlling robotic devices. Human-machine interaction is a key component of the control infrastructure. The most common solution is mobile devices or embedded touch screens, as well as next generation virtual reality devices. In human-machine interaction, most input devices are controlled manually, which is not always convenient, and sometimes even impossible. One option is gesture control, which has become increasingly common in the last few years. This artificial cognitive "sensory perception" or ability is a communication channel between a human and a machine. This article presents a two-steps approach to real-time control robotic devices. The first step is the hand recognition method base on palm detection (SSD Detector) and hand landmark models. After a palm detection, the hand landmark model performs fine localization of the key points of the 3-D coordinate of the hand inside the detected areas of the hand through regression and direct coordinate prediction. The model learns a consistent internal representation of the hand posture and is resistant to even partially visible hands and self-occlusions. The second human gesture recognition step is based on obtaining the coordinates of the hand, the distance from the camera to the hand in space, raised, lowered fingers and other indicators that allow you to accurately determine the shown gesture. In terms of gesture recognition accuracy, the proposed real-time system is better than the state-of-the-art methods.

show abstract

Action recognition algorithm from visual sensor data for contactless robot control systems

Cited by 3 publications

References 20 publications

Action recognition algorithm from multimodal images for human-robot collaboration systems

Action recognition algorithm from multimodal images for human-robot collaboration systems

A multimodal visual guided robot collaborative system based on the classification of multi-class human motion

Real-time robotic hand control using human gesture recognition

Contact Info

Product

Resources

About