Fusion Machine Learning Strategies for Multi-modal Sensor-based Hand Gesture Recognition

Abstract

Hand gesture recognition has attracted the attention of many scientists, because of its high applicability in fields such as sign language expression and human machine interaction. Many approaches have been deployed to detect and recognize hand gestures, like wearable devices, image information, and/or a combination of sensors and computer vision. However, the method of using wearable sensors brings much higher accuracy and is less affected by occlusion, lighting conditions, and complex background. Existing solutions separately utilize sensor information and/or only use sensor information processing and decision-making algorithms over conventional threshold comparison algorithms and do not analyze data or utilize machine learning algorithms. In this paper, a multi-modal solution is proposed that combines information for measuring the curvature of the fingers and sensors for measuring angular velocity and acceleration. The provided information from the sensors is normalized and analyzed and various fusion strategies are used. Then, the most suitable algorithm for these sensor-based multiple modalities is proposed. The proposed system also analyzes the differences between gestures and actions that are almost similar but in fact, they are just normal moving gestures.

RIOD:Reinforced Image-based Object Detection for Unruly Weather Conditions

“…In [10], a plug-and-play module, called cross-fusion, was added to the YOLOv5 model to combine the features of multiple convolutional layers, but its improved performance came at the cost of increased computational complexity. In [11][12], a multimodal sensor fusion technique was proposed, which presented an enhanced object detection rate for autonomous vehicles. However, the integration of four different sensors and the simultaneous processing of their outputs in real time proves to be cumbersome and expensive compared to other methods.…”

“…Rather than complicating the design of Deep Neural Networks (DNN) [9][10][11][12][18][19], a separate filter was developed to be placed between the camera and the CNN module. This filter mechanism enriches the image captured by the camera and feeds it to CNN models, as shown in Figure 2.…”