Abnormal gesture recognition based on multi-model fusion strategy

Lin, Chao; Lin, Xuxin; Xie, Yiliang; Liang, Yanyan

doi:10.1007/s00138-018-0969-0

Cited by 8 publications

(2 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Compared to ResC3D [17], [48], our model has significant room for improvement. So far, we have only used a simple VGG16 network in the dual stream network section.…”

Section: Evaluation On Isogdmentioning

confidence: 99%

Multimodal Spatiotemporal Networks for Sign Language Recognition

Zhang

Meng

et al. 2019

IEEE Access

View full text Add to dashboard Cite

Different from other human behaviors, sign language has the characteristics of limited local motion of upper limb and meticulous hand action. Some sign language gestures are ambiguous in RGB video due to the influence of lighting and background color, which affects the recognition accuracy. We propose a multimodal deep learning architecture for sign language recognition which effectively combines RGB-D input and two-stream spatiotemporal networks. Depth videos, as an effective compensation of RGB input, can supply additional distance information about the signer's hands. A novel sampling method called ARSS (Aligned Random Sampling in Segments) is put forward to select and align optimal RGB-D video frames, which improves the capacity utilization of multimodal data and reduces the redundancy. We get the hand ROI by joints information of RGB data for local focus in spatial stream. D-shift Net is proposed as depth motion feature extraction in temporal stream, which fully utilizes three dimensional motion information of the sign language. Both streams are fused by convolutional fusion layer to get complementary features. Our approach explored the multimodal information and enhanced the recognition precision. It obtains the state-the-of-art performance on the datasets of CSL (96.7%) and IsoGD (63.78%). INDEX TERMS Sign language recognition, two-stream network, motion features, multimodal data.

show abstract

“…Compared to ResC3D [17], [48], our model has significant room for improvement. So far, we have only used a simple VGG16 network in the dual stream network section.…”

Section: Evaluation On Isogdmentioning

confidence: 99%

Multimodal Spatiotemporal Networks for Sign Language Recognition

Zhang

Meng

et al. 2019

IEEE Access

View full text Add to dashboard Cite

show abstract

“…They conducted an experimental evaluation using the Chalearn IsoGD Dataset [15] showing the effectiveness of the attention model. In the same line, Lin et al, in their Abnormal Gesture Recognition Based on Multi-Model Fusion Strategy article [10], proposed a novel refined fused model that combines a masked (RGB and Depth) ResC3D networks with a skeleton LSTM for abnormal gesture recognition in RGBD videos. Based on their experimental results, their proposed method could distinguish the abnormal gesture samples effectively achieving state-of-the-art performance in the IsoGD dataset.…”

mentioning

confidence: 99%