Curve Matching from the View of Manifold for Sign Language Recognition

Lin, Yuan; Chai, Xiujuan; Zhou, Yu; Chen, Xilin

doi:10.1007/978-3-319-16634-6_18

Cited by 12 publications

(6 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Oliveira et al [18] used two handed data gloves to capture sign language movements and fed them into neural networks for recognition, achieving recognition of English words. Lin et al [19] used cameras to obtain data on people wearing colored gloves and performed data preprocessing such as color segmentation on these image data. Although sensor based sign language recognition has made significant progress, these devices require sign language performers to comply with specific wearing requirements, making the entire process cumbersome.…”

Section: Related Workmentioning

confidence: 99%

Video-Based Sign Language Recognition via ResNet and LSTM Network

Huang,

Chouvatut

2024

Preprint

View full text Add to dashboard Cite

Sign language recognition technology can help people with hearing impairments to communicate with those who are hearing impaired. At present, with the rapid development of society, deep learning technology also provided certain technical support for sign language recognition work. In sign language recognition tasks, the use of traditional convolutional neural networks to extract spatio-temporal features from sign language videos suffers from insufficient feature extraction, resulting in low recognition rates. Nevertheless, video-based sign language datasets are very large and require a lot of computational resources for training and generalisation must be ensured, which poses a challenge for recognition. In this paper, we have presented a video-based sign language recognition method based on resnet and lstm. As the number of network layers increases, the ResNet network can effectively solve the granularity explosion problem and obtain better time series features. We use the Resnet convolutional network as the backbone model. At the initialisation stage, we obtain sign language features using ResNet; then, the learned feature space is used as the input of LSTM network to obtain long sequence features. The experimental results show that the accuracy of the above model is better than the mainstream model, and it can effectively extract the spatio-temporal features in sign language videos and improve the recognition rate of sign language actions.

show abstract

Section: Related Workmentioning

confidence: 99%

Video-Based Sign Language Recognition via ResNet and LSTM Network

Huang,

Chouvatut

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…TH refers to a virtual or digital representation of the human face or head, typically used in multimedia applications, computer [78] 2012 ∼3K Video-Text English Link SIGNUM [79] 2013 ∼33K Video-Text Germany Link DEVISIGN [80] 2014 ∼24k Video-Text Chinese Link ASL-LEX 1.0 [81] 2017 ∼1K Video-Text English Link PHOENIX14T [82] 2018 ∼68K Video-Text Germany Link CMLR [83] 2019 ∼102K Image-Text Chinese Link KETI [84] 2019 ∼15K Video-Text Korean Not Available GSL [85] 2020 ∼3K Video-Text Greek Link ASL-LEX 2.0 [86] 2021 ∼ 10K Video-Text-Depth English Link How2sign [87] 2021 ∼35K Video-Text-Skelton(2D)-Depth English Link Slovo [88] 2023 ∼20K Video-Text Russian Link AASL [89] 2023 ∼8K Image-Text Arabic Link ASL-27C [83] 2023 ∼23K Image-Text English Link Cued Speech FCS [90] 2018 ∼13k Video-Text-Audio French Link BEC [59] 2019 ∼3k Video-Text-Audio English Link PCSC [91] 2020 20 (P) Video-Text-Audio Polish Link CLeLfPC [92] 2022 350 Video-Text-Audio French Link MCCS-2023 [41] 2023 ∼132k Video-Text-Audio-Skelton(2,3D) Chinese Link…”

Section: Talking Headmentioning

confidence: 99%

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Liu

Wang

et al. 2021

Front. Neurorobot.

View full text Add to dashboard Cite

The redundant information, noise data generated in the process of single-modal feature extraction, and traditional learning algorithms are difficult to obtain ideal recognition performance. A multi-modal fusion emotion recognition method for speech expressions based on deep learning is proposed. Firstly, the corresponding feature extraction methods are set up for different single modalities. Among them, the voice uses the convolutional neural network-long and short term memory (CNN-LSTM) network, and the facial expression in the video uses the Inception-Res Net-v2 network to extract the feature data. Then, long and short term memory (LSTM) is used to capture the correlation between different modalities and within the modalities. After the feature selection process of the chi-square test, the single modalities are spliced to obtain a unified fusion feature. Finally, the fusion data features output by LSTM are used as the input of the classifier LIBSVM to realize the final emotion recognition. The experimental results show that the recognition accuracy of the proposed method on the MOSI and MELD datasets are 87.56 and 90.06%, respectively, which are better than other comparison methods. It has laid a certain theoretical foundation for the application of multimodal fusion in emotion recognition.

show abstract

“…Since the three-dimensional convolutional neural network only reduces the image in a small proportion in the spatial domain, and the convolutional long short-term memory recursive network does not change the spatial size of the feature map, the final long-term spatiotemporal feature vector has a relatively high spatial size (the size of the long-term spatiotemporal eigenvector is 28×28 in our network due to the input size of the 3D convolutional neural network is 112 × 112). A spatial pyramid pool [20] was inserted between the convolutional long short-term memory recurrent network and the fully connected layer to reduce the size. Therefore the final fully connected layer can have fewer parameters.…”

Section: Architecture Of Our Feature Extraction Networkmentioning

confidence: 99%

Sign language recognition based on skeleton with K3D-LSTMresidual networks

Han

Huangfu

Min

et al. 2021

Preprint

View full text Add to dashboard Cite

Most existing deep learning-based dynamic sign language recognition methods directly use either the video sequence based on RGB information, or whole sequences instead of only the video sequence that represents the change of gesture. These characteristics lead to inaccurate extraction of hand gesture features and failure to achieve good recognition accuracy for complex gestures. In order to solve these problems, this paper proposes a new method of dynamic hand gesture recognition for key skeleton information, which combines residual convolutional neural network and long short-term memory recurrent network, which is called KLSTM-3D residual network (K3D ResNet). In K3DResNet, the spatiotemporal complexity of network computation is reduced by extracting the representative skeleton frame of gesture change. Then, the spatiotemporal features are extracted from the skeleton keyframe sequence, and the intermediate score corresponding to each action in the video sequence is established after the feature analysis. Finally, the classification of video sequences can accurately identify sign language. Experiments were performed on datasets DHG14/28 and SHREC’17 Track. The accuracy of verification on dataset DEVISIGN D reached 88.6%. In addition, the accuracy of the combination of RGB and skeleton information reached 93.2%.

show abstract

Curve Matching from the View of Manifold for Sign Language Recognition

Cited by 12 publications

References 21 publications

Video-Based Sign Language Recognition via ResNet and LSTM Network

Video-Based Sign Language Recognition via ResNet and LSTM Network

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Sign language recognition based on skeleton with K3D-LSTMresidual networks

Contact Info

Product

Resources

About