Sign Pose-based Transformer for Word-level Sign Language Recognition

Boháček, Matyáš; Hrúz, Marek

doi:10.1109/wacvw54805.2022.00024

Cited by 82 publications

(40 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In this section, we describe the individual classification methods, which were used in this paper. We utilize three types of classification models in total-I3D [6,40], Times-Former [41], and SPOTER [10]. These specific models were chosen with variability in mind.…”

Section: Classification Methodsmentioning

confidence: 99%

“…Inspired by Yan et al [26], who used a spatio-temporal graph convolutional network (GCN) for action recognition, Vázquez-Enríquez et al [27] used GCNs also for SLR. Last year, Boháček et al [10] introduced pose-based transformer SPOTER and reached very promising results. In small training data protocol, SPOTER outperforms even the visual-based approaches.…”

Section: Sign Language Recognitionmentioning

confidence: 99%

“…The SPOTER architecture was implemented using PyTorch. We followed the training setup from the original paper [10]. We hence used the SGD optimizer with a learning rate of 0.001.…”

Section: Spotermentioning

confidence: 99%

“…In general, when a large amount of training data and computational power is available, the appearance-based methods outperform the pose-based methods. If the conditions are not met, one should utilize the pose-based approach, as was shown, for example, in [10].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

One Model is Not Enough: Ensembles for Isolated Sign Language Recognition

Hrúz

Gruber

Kanis

et al. 2022

Sensors

View full text Add to dashboard Cite

In this paper, we dive into sign language recognition, focusing on the recognition of isolated signs. The task is defined as a classification problem, where a sequence of frames (i.e., images) is recognized as one of the given sign language glosses. We analyze two appearance-based approaches, I3D and TimeSformer, and one pose-based approach, SPOTER. The appearance-based approaches are trained on a few different data modalities, whereas the performance of SPOTER is evaluated on different types of preprocessing. All the methods are tested on two publicly available datasets: AUTSL and WLASL300. We experiment with ensemble techniques to achieve new state-of-the-art results of 73.84% accuracy on the WLASL300 dataset by using the CMA-ES optimization method to find the best ensemble weight parameters. Furthermore, we present an ensembling technique based on the Transformer model, which we call Neural Ensembler.

show abstract

Section: Classification Methodsmentioning

confidence: 99%

Section: Sign Language Recognitionmentioning

confidence: 99%

“…The SPOTER architecture was implemented using PyTorch. We followed the training setup from the original paper [10]. We hence used the SGD optimizer with a learning rate of 0.001.…”

Section: Spotermentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

One Model is Not Enough: Ensembles for Isolated Sign Language Recognition

Hrúz

Gruber

Kanis

et al. 2022

Sensors

View full text Add to dashboard Cite

show abstract

“…Achieve accuracy 96.66% for 12-word recognition. Boháček and Hrúz [148] performed isolated sign language recognition using SPOTER (Sign pose base transformer) validated with LSA64 and WLASL datasets, resulting in a 100% accuracy for LSA64 and 63.18% and 43.78% accuracy for WLASL 100 and WLASL 300, respectively. The current state-of-the-art SLR model is summarized in a Table XI for better understanding.…”

Section: B Study Of Current State-of-the-art Models For Sign Language...mentioning

confidence: 99%

A Comprehensive Review of Sign Language Recognition: Different Types, Modalities, and Datasets

Madhiarasan¹,

Roy²

2022

Preprint

View full text Add to dashboard Cite

A machine can understand human activities, and the meaning of signs can help overcome the communication barriers between the inaudible and ordinary people. Sign Language Recognition (SLR) is a fascinating research area and a crucial task concerning computer vision and pattern recognition. Recently, SLR usage has increased in many applications, but the environment, background image resolution, modalities, and datasets affect the performance a lot. Many researchers have been striving to carry out generic real-time SLR models. This review paper facilitates a comprehensive overview of SLR and discusses the needs, challenges, and problems associated with SLR. We study related works about manual and non-manual, various modalities, and datasets. Research progress and existing stateof-the-art SLR models over the past decade have been reviewed. Finally, we find the research gap and limitations in this domain and suggest future directions. This review paper will be helpful for readers and researchers to get complete guidance about SLR and the progressive design of the state-of-the-art SLR model.

show abstract