PiSLTRc: Position-Informed Sign Language Transformer With Content-Aware Convolution

Xie, Pan; Zhao, Mengyi; Hu, Xiaohui

doi:10.1109/tmm.2021.3109665

Cited by 28 publications

(12 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…SL-Transf (Camgoz et al, 2020b) models utilize pre-trained features from CNN-LSTM-HMM and jointly learn sign language recognition and translation. PiSLTRc (Xie et al, 2021) uses positioninformed temporal convolution base on SL-Transf. SignBT (Zhou et al, 2021a) uses Sign Back-Translation for data augmentation.…”

Section: Comparison Resultsmentioning

confidence: 99%

“…carefully design and test different data augmentations and combinations to learn a noiseinvariant sentence representation. mRASP2 model (Pan et al, 2021) leverages monolingual data and bilingual data under a unified training framework to close the representation gap of different languages by introducing contrastive learning and aligned augmentation. Gao et al (2021) apply the dropout (Srivastava et al, 2014) twice acting as minimal data augmentation to obtain two different embeddings as "positive pairs" for each sentence.…”

Section: Contrastive Learningmentioning

confidence: 99%

“…Recent works Pan et al, 2021;Qin et al, 2021) also demonstrate that contrastive learning has tremendous potential in lowresource scenarios, especially for NMT. Pan et al (2021) proposes a multilingual contrastive learning framework mRASP2 for multilingual NMT which achieves significant improvements (10.26 BLEU) on the 30 zero-shot translation directions.…”

Section: Introductionmentioning

confidence: 96%

“…Currently, contrastive learning has become a very popular technique in computer vision and natural language processing community with solid performance (Chen et al, 2020a;He et al, 2020;Chen et al, 2020b;Fang et al, 2020;Carlsson et al, 2020;Giorgi et al, 2020;Yan et al, 2021). Recent works Pan et al, 2021;Qin et al, 2021) also demonstrate that contrastive learning has tremendous potential in lowresource scenarios, especially for NMT. Pan et al (2021) proposes a multilingual contrastive learning framework mRASP2 for multilingual NMT which achieves significant improvements (10.26 BLEU) on the 30 zero-shot translation directions.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Token-level Contrastive Framework for Sign Language Translation

Fu¹,

Peigen²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Sign language translation (SLT) is an important technology that can bridge the communication gap between the deaf and the hearing people. SLT task is essentially a low-resource problem due to the scarcity of publicly available parallel data. To this end, inspired by the success of neural machine translation methods based on contrastive learning, we propose ConSLT, a novel token-level Contrastive learning framework for Sign Language Translation. Unlike previous contrastive learning based works whose goal is to obtain better sentence representation, ConSLT aims to learn effective token representation by pushing apart tokens from different sentences. Concretely, our model follows the two-stage SLT method. First, in the recoginition stage, we use a stateof-the-art continuous sign language recognition model to recognize glosses from sign frames. Then, in the translation stage, we adopt the Transformer framework while introducing contrastive learning. Specifically, we pass each sign glosses to the Transformer model twice to obtain two different hidden layer representations for each token as "positive examples" and randomly sample K tokens that are not in the current sentence from the vocabulary as "negative examples" for each token. Experimental results demonstrate that ConSLT achieves new state-of-the-art performance on PHOENIX14T dataset, with +1.48 BLEU improvements.

show abstract

Section: Comparison Resultsmentioning

confidence: 99%

Section: Contrastive Learningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 96%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Token-level Contrastive Framework for Sign Language Translation

Fu¹,

Peigen²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Finally, the proposed multi-level CTC loss part is used for training to obtain recognition results. The research in this paper is based on 1DCNN, using temporal receptive fields of different scales to enhance sequence modeling capability, followed by the self-attention mechanism of transfomers [21,22] to better access long-distance dependent information and improve the discriminative power of features. Finally using the proposed multilevel CTC loss, which not only can better decode the temporal features, but also can make the parameters of the shallow network well updated, and then efficiently train the frame-wise feature extraction network and temporal modeling network, further improving the recognition performance.…”

Section: Introductionmentioning

confidence: 99%

Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition

Zhou

et al. 2020

AAAI

150

View full text Add to dashboard Cite

Despite the recent success of deep learning in continuous sign language recognition (CSLR), deep models typically focus on the most discriminative features, ignoring other potentially non-trivial and informative contents. Such characteristic heavily constrains their capability to learn implicit visual grammars behind the collaboration of different visual cues (i,e., hand shape, facial expression and body posture). By injecting multi-cue learning into neural network design, we propose a spatial-temporal multi-cue (STMC) network to solve the vision-based sequence learning problem. Our STMC network consists of a spatial multi-cue (SMC) module and a temporal multi-cue (TMC) module. The SMC module is dedicated to spatial representation and explicitly decomposes visual features of different cues with the aid of a self-contained pose estimation branch. The TMC module models temporal correlations along two parallel paths, i.e., intra-cue and inter-cue, which aims to preserve the uniqueness and explore the collaboration of multiple cues. Finally, we design a joint optimization strategy to achieve the end-to-end sequence learning of the STMC network. To validate the effectiveness, we perform experiments on three large-scale CSLR benchmarks: PHOENIX-2014, CSL and PHOENIX-2014-T. Experimental results demonstrate that the proposed method achieves new state-of-the-art performance on all three benchmarks.

show abstract

Efficient Continuous Sign Language Recognition with Temporal Shift and Channel Attention

Nam,

Takahashi

2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

PiSLTRc: Position-Informed Sign Language Transformer With Content-Aware Convolution

Cited by 28 publications

References 37 publications

A Token-level Contrastive Framework for Sign Language Translation

A Token-level Contrastive Framework for Sign Language Translation

Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition

Efficient Continuous Sign Language Recognition with Temporal Shift and Channel Attention

Contact Info

Product

Resources

About