Sensor fusion using EMG and vision for hand gesture classification in mobile applications

Ceolini, Enea; Taverni, Gemma; Khacef, Lyes; Payvand, Melika; Donati, Elisa

doi:10.1109/biocas.2019.8919210

Cited by 11 publications

(9 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this paper we present a fully-neuromorphic implementation of sensor fusion for hand-gesture recognition. The proposed work is based on a previous work of sensor fusion for hand-gesture recognition, using standard machine learning approaches implemented in a cell phone application for personalized medicine (Ceolini et al, 2019b ). The paper showed how a CNN performed better, in terms of accuracy, than a Support Vector Machine (SVM) on the hand-gesture recognition task.…”

Section: Introductionmentioning

confidence: 99%

Hand-Gesture Recognition Based on EMG and Event-Based Camera Sensor Fusion: A Benchmark in Neuromorphic Computing

et al. 2020

Self Cite

View full text Add to dashboard Cite

Hand gestures are a form of non-verbal communication used by individuals in conjunction with speech to communicate. Nowadays, with the increasing use of technology, hand-gesture recognition is considered to be an important aspect of Human-Machine Interaction (HMI), allowing the machine to capture and interpret the user's intent and to respond accordingly. The ability to discriminate between human gestures can help in several applications, such as assisted living, healthcare, neuro-rehabilitation, and sports. Recently, multi-sensor data fusion mechanisms have been investigated to improve discrimination accuracy. In this paper, we present a sensor fusion framework that integrates complementary systems: the electromyography (EMG) signal from muscles and visual information. This multi-sensor approach, while improving accuracy and robustness, introduces the disadvantage of high computational cost, which grows exponentially with the number of sensors and the number of measurements. Furthermore, this huge amount of data to process can affect the classification latency which can be crucial in real-case scenarios, such as prosthetic control. Neuromorphic technologies can be deployed to overcome these limitations since they allow real-time processing in parallel at low power consumption. In this paper, we present a fully neuromorphic sensor fusion approach for hand-gesture recognition comprised of an event-based vision sensor and three different neuromorphic processors. In particular, we used the event-based camera, called DVS, and two neuromorphic platforms, Loihi and ODIN + MorphIC. The EMG signals were recorded using traditional electrodes and then converted into spikes to be fed into the chips. We collected a dataset of five gestures from sign language where visual and electromyography signals are synchronized. We compared a fully neuromorphic approach to a baseline implemented using traditional machine learning approaches on a portable GPU system. According to the chip's constraints, we designed specific spiking neural networks (SNNs) for sensor fusion that showed classification accuracy comparable to the software baseline. These neuromorphic alternatives have increased inference time, between 20 and 40%, Ceolini et al. Sensor Fusion Neuromorphic Benchmark with respect to the GPU system but have a significantly smaller energy-delay product (EDP) which makes them between 30× and 600× more efficient. The proposed work represents a new benchmark that moves neuromorphic computing toward a real-world scenario.

show abstract

Section: Introductionmentioning

confidence: 99%

Hand-Gesture Recognition Based on EMG and Event-Based Camera Sensor Fusion: A Benchmark in Neuromorphic Computing

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Recently, with the vigorous development of computer vision, some researchers are trying combining the vision and EMG information for limb motion classification task. In a newly published work [ 113 ], researches implemented a framework that allows the integration of multi-sensors, EMG and visual information, to perform sensor fusion and to improve the accuracy of hand gesture recognition tasks. For embedded applications, even-based cameras were utilized to run on the limited computational resources of mobile phones.…”

Section: Multisensory Fusionmentioning

confidence: 99%

EMG-Centered Multisensory Based Technologies for Pattern Recognition in Rehabilitation: State of the Art and Challenges

et al. 2020

View full text Add to dashboard Cite

In the field of rehabilitation, the electromyography (EMG) signal plays an important role in interpreting patients’ intentions and physical conditions. Nevertheless, utilizing merely the EMG signal suffers from difficulty in recognizing slight body movements, and the detection accuracy is strongly influenced by environmental factors. To address the above issues, multisensory integration-based EMG pattern recognition (PR) techniques have been developed in recent years, and fruitful results have been demonstrated in diverse rehabilitation scenarios, such as achieving high locomotion detection and prosthesis control accuracy. Owing to the importance and rapid development of the EMG centered multisensory fusion technologies in rehabilitation, this paper reviews both theories and applications in this emerging field. The principle of EMG signal generation and the current pattern recognition process are explained in detail, including signal preprocessing, feature extraction, classification algorithms, etc. Mechanisms of collaborations between two important multisensory fusion strategies (kinetic and kinematics) and EMG information are thoroughly explained; corresponding applications are studied, and the pros and cons are discussed. Finally, the main challenges in EMG centered multisensory pattern recognition are discussed, and a future research direction of this area is prospected.

show abstract

“…Indeed, the discrimination of human gestures using wearable solutions is extremely important as a supporting technique for assisted living, healthcare of the elderly and neuro-rehabilitation. For this purpose, we proposed in References [89,90] a framework that allows the integration of multi-sensory data to perform sensor fusion based on supervised learning. The framework was applied for the hand gestures recognition task with five hand gestures: Pinky (P), Elle (E), Yo (Y), Index (I) and Thumb (T).…”

Section: Dvs/emg Hand Gestures Databasementioning

confidence: 99%

“…The frames were generated by counting the events occurring in a fixed time window for each of the pixels separately, followed by a min-max normalization to get gray scale frames. The time window was fixed to 200 ms so that the DVS frames can be synchronized with the EMG signal, as further detailed in Reference [89]. The event frames obtained from the DVS camera have a resolution of 128 × 128 pixels.…”

Section: Dvs Hand Gesturesmentioning

confidence: 99%

“…For the EMG signal, we selected two time domain features that are commonly used in the literature [102]: the Mean Absolute Value (MAV) and the Root Mean Square (RMS) which are calculated over the same window of length 20 ms, as detailed in Reference [89]. With the same strategy as for DVS frames, we extract CNN-based features of 192 dimensions.…”

Section: Emg Hand Gesturesmentioning

confidence: 99%

See 1 more Smart Citation

Brain-Inspired Self-Organization with Cellular Neuromorphic Computing for Multimodal Unsupervised Learning

2020

Self Cite

View full text Add to dashboard Cite

Cortical plasticity is one of the main features that enable our ability to learn and adapt in our environment. Indeed, the cerebral cortex self-organizes itself through structural and synaptic plasticity mechanisms that are very likely at the basis of an extremely interesting characteristic of the human brain development: the multimodal association. In spite of the diversity of the sensory modalities, like sight, sound and touch, the brain arrives at the same concepts (convergence). Moreover, biological observations show that one modality can activate the internal representation of another modality when both are correlated (divergence). In this work, we propose the Reentrant Self-Organizing Map (ReSOM), a brain-inspired neural system based on the reentry theory using Self-Organizing Maps and Hebbian-like learning. We propose and compare different computational methods for unsupervised learning and inference, then quantify the gain of the ReSOM in a multimodal classification task. The divergence mechanism is used to label one modality based on the other, while the convergence mechanism is used to improve the overall accuracy of the system. We perform our experiments on a constructed written/spoken digits database and a Dynamic Vision Sensor (DVS)/EletroMyoGraphy (EMG) hand gestures database. The proposed model is implemented on a cellular neuromorphic architecture that enables distributed computing with local connectivity. We show the gain of the so-called hardware plasticity induced by the ReSOM, where the system’s topology is not fixed by the user but learned along the system’s experience through self-organization.

show abstract

Sensor fusion using EMG and vision for hand gesture classification in mobile applications

Cited by 11 publications

References 14 publications

Hand-Gesture Recognition Based on EMG and Event-Based Camera Sensor Fusion: A Benchmark in Neuromorphic Computing

Hand-Gesture Recognition Based on EMG and Event-Based Camera Sensor Fusion: A Benchmark in Neuromorphic Computing

EMG-Centered Multisensory Based Technologies for Pattern Recognition in Rehabilitation: State of the Art and Challenges

Brain-Inspired Self-Organization with Cellular Neuromorphic Computing for Multimodal Unsupervised Learning

Contact Info

Product

Resources

About