Multimodal Gesture Recognition Using Multi-stream Recurrent Neural Network

Nishida, Noriki; Nakayama, Hideki

doi:10.1007/978-3-319-29451-3_54

Cited by 40 publications

(24 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This study explored the use of Long-term Recurrent Convolutional Network (LRCN) for a variety of vision tasks including activity recognition based on video input for this comparison (Donahue et al, 2017). Other studies have also employed long short-term memory (LSTM) for gesture recognition for its capability to learn activities of varying time length (Nishida and Nakayama, 2016;Tsironi et al, 2016).…”

Section: Application Backgroundmentioning

confidence: 99%

Investigation of Channel Selection for Gesture Classification for Prosthesis Control Using Force Myography: A Case Study

Ahmadizadeh

Pousett²,

Menon

2019

Front. Bioeng. Biotechnol.

View full text Add to dashboard Cite

Background: Various human machine interfaces (HMIs) are used to control prostheses, such as robotic hands. One of the promising HMIs is Force Myography (FMG). Previous research has shown the potential for the use of high density FMG (HD-FMG) that can lead to higher accuracy of prosthesis control. Motivation: The more sensors used in an FMG controlled system, the more complicated and costlier the system becomes. This study proposes a design method that can produce powered prostheses with performance comparable to that of HD-FMG controlled systems using a fewer number of sensors. An HD-FMG apparatus would be used to collect information from the user only in the design phase. Channel selection would then be applied to the collected data to determine the number and location of sensors that are vital to performance of the device. This study assessed the use of multiple channel selection (CS) methods for this purpose. Methods: In this case study, three datasets were used. These datasets were collected from force sensitive resistors embedded in the inner socket of a subject with transradial amputation. Sensor data were collected as the subject carried out five repetitions of six gestures. Collected data were then used to asses five CS methods: Sequential forward selection (SFS) with two different stopping criteria, minimum redundancy-maximum relevance (mRMR), genetic algorithm (GA), and Boruta. Results: Three out of the five methods (mRMR, GA, and Boruta) were able to decrease channel numbers significantly while maintaining classification accuracy in all datasets. Neither of them outperformed the other two in all datasets. However, GA resulted in the smallest channel subset in all three of the datasets. The three selected methods were also compared in terms of stability [i.e., consistency of the channel subset chosen by the method as new training data were introduced or some training data were removed (Chandrashekar and Sahin, 2014)]. Boruta and mRMR resulted in less instability compared to GA when applied to the datasets of this study. Conclusion: This study shows feasibility of using the proposed design method that can produce prosthetic systems that are simpler than HD-FMG systems but have performance comparable to theirs.

show abstract

Section: Application Backgroundmentioning

confidence: 99%

Investigation of Channel Selection for Gesture Classification for Prosthesis Control Using Force Myography: A Case Study

Ahmadizadeh

Pousett²,

Menon

2019

Front. Bioeng. Biotechnol.

View full text Add to dashboard Cite

show abstract

“…CLDNN [25] analyzed the effect of adding CNN layers to LSTM and LRCN [3] extended the convolutional LSTM for visual recognition problems. Besides, multiple-stream based methods have been proposed for tackling the multiple modality data [20,27], which separately learn the spatial features from video frames and the temporal feature from the dense optical flow for action recognition.…”

Section: Gesture Recognition Using the Sequential Video Sequencesmentioning

confidence: 99%

LE-HGR: A Lightweight and Efficient RGB-Based Online Gesture Recognition Network for Embedded AR Devices

Xie

Wang

Shao

et al. 2019

2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)

View full text Add to dashboard Cite

Figure 1: LE-HGR Framework. The overall framework of the proposed gesture recognition algorithm. The framework contains four sub-modules: Hand candidate detector, multi-task CNN, hand trace mapping, and trace sequence network. ABSTRACTOnline hand gesture recognition (HGR) techniques are essential in augmented reality (AR) applications for enabling natural humanto-computer interaction and communication. In recent years, the consumer market for low-cost AR devices has been rapidly growing, while the technology maturity in this domain is still limited. Those devices are typical of low prices, limited memory, and resourceconstrained computational units, which makes online HGR a challenging problem. To tackle this problem, we propose a lightweight and computationally efficient HGR framework, namely LE-HGR, to enable real-time gesture recognition on embedded devices with low computing power. We also show that the proposed method is of high accuracy and robustness, which is able to reach high-end performance in a variety of complicated interaction environments.To achieve our goal, we first propose a cascaded multi-task convolutional neural network (CNN) to simultaneously predict probabilities of hand detection and regress hand keypoint locations online. We show that, with the proposed cascaded architecture design, false-positive estimates can be largely eliminated. Additionally, an associated mapping approach is introduced to track the hand trace via the predicted locations, which addresses the interference of multi-handedness. Subsequently, we propose a trace sequence neural network (TraceSeqNN) to recognize the hand gesture by exploiting the motion features of the tracked trace. Finally, we provide a variety of experimental results to show that the proposed framework is able to achieve state-of-the-art accuracy with significantly reduced computational cost, which are the key properties for enabling real-time applications in low-cost commercial devices such as mobile devices and AR/VR headsets. * These authors contributed equally to this work.

show abstract

“…Therefore the position data of the joint cannot be correctly calculated. To prevent the situation, the hand position is confirmed using the skin color [6].…”

Section: Acquisition Of Hand Position Datamentioning

confidence: 99%

“…This study had the disadvantage of recognizing only the still hand shape. Nishida and Nakayama [6] constructed a multistream recurrent neural network composed of several stages of long short-term memory (LSTM) and recognized hand movements. Though it showed 97.8% recognition ratio, it had serious limitation.…”

mentioning

confidence: 99%

Recognition of Natural Hand Gestures Using Bidirectional Long Short-Term Memory Model

Kim¹,

Rhee²

2018

IJFIS

View full text Add to dashboard Cite

Recent studies have been trying to use voices and body gestures similar to human communication methods to control intelligent devices such as robots instead of giving commands using a keyboard or a mouse. Human being obtains about 80% of information using vision, and about 55% of all meaning of communication is related to vision. In addition, hand gestures are most frequently used in non-verbal communication. Therefore, there have been many researches to give commands to a robot. Existing studies were limited to recognition of fixed hand gestures and shapes, requiring users to be educated on motions that can be used to communicate with robots. To solve this problem, we use fuzzy inference to select meaningful gesture in continuous gesture at first. Hand position is interpolated using the Lagrangian method. Also, we use the Kalman filter for object-occlusion or self-occlusion. After generating a sequence of continuously received hand positions using a chain code, the fuzzy theory is used to select meaningful motions among various hand gestures. Eventually, we perform a method of recognizing meaningful hand gestures using a recurrent neural network with bidirectional long short-term memory (LSTM) architecture. Even though selecting the meaningful motion was hard, if it was selected correctly, the ratio of hand gesture recognition was very high in experiment results.

show abstract

Multimodal Gesture Recognition Using Multi-stream Recurrent Neural Network

Cited by 40 publications

References 19 publications

Investigation of Channel Selection for Gesture Classification for Prosthesis Control Using Force Myography: A Case Study

Investigation of Channel Selection for Gesture Classification for Prosthesis Control Using Force Myography: A Case Study

LE-HGR: A Lightweight and Efficient RGB-Based Online Gesture Recognition Network for Embedded AR Devices

Recognition of Natural Hand Gestures Using Bidirectional Long Short-Term Memory Model

Contact Info

Product

Resources

About