A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation

Chen, Yutong; Wei, Fangyun; Sun, Xiaofeng; Wu, Zhirong; Lin, Stephen

doi:10.48550/arxiv.2203.04287

Cited by 2 publications

(2 citation statements)

References 30 publications

(90 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…representations to predict the action category. In contrast, many practical applications, e.g., sign language translation [4,5,13], robotic imitation learning [29,36], action alignment [6,21,23] and phase classification [16,27,37,50] require algorithms having ability to model long videos with hundreds of frames and extract frame-wise representations rather than the global features (Fig. 1).…”

Section: Liquid Starts Exitingmentioning

confidence: 99%

Frame-wise Action Representations for Long Videos via Sequence Contrastive Learning

Chen¹,

Wei²,

Li³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Prior works on action representation learning mainly focus on designing various architectures to extract the global representations for short video clips. In contrast, many practical applications such as video alignment have strong demand for learning dense representations for long videos. In this paper, we introduce a novel contrastive action representation learning (CARL) framework to learn frame-wise action representations, especially for long videos, in a selfsupervised manner. Concretely, we introduce a simple yet efficient video encoder that considers spatio-temporal context to extract frame-wise representations. Inspired by the recent progress of self-supervised learning, we present a novel sequence contrastive loss (SCL) applied on two correlated views obtained through a series of spatio-temporal data augmentations. SCL optimizes the embedding space by minimizing the KL-divergence between the sequence similarity of two augmented views and a prior Gaussian distribution of timestamp distance. Experiments on FineGym, PennAction and Pouring datasets show that our method outperforms previous state-of-the-art by a large margin for downstream fine-grained action classification. Surprisingly, although without training on paired videos, our approach also shows outstanding performance on video alignment and fine-grained frame retrieval tasks. Code and models are available at https://github.com/ minghchen/CARL_code.

show abstract

Section: Liquid Starts Exitingmentioning

confidence: 99%

Frame-wise Action Representations for Long Videos via Sequence Contrastive Learning

Chen¹,

Wei²,

Li³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Authors of [16] apply transfer learning for end-to-end sign language translation on two different datasets. They used GPT2 pre-trained model on the German Sign Language corpus and got remarkable results.…”

Section: Related Workmentioning

confidence: 99%

Transfer learning for Azerbaijani Sign Language Recognition

Abdullayeva¹,

Alishzade²

2022

Informatics and Control Problems

View full text Add to dashboard Cite

The goal of sign language technologies is to develop a bridging solution for the communication gap between the hearing-impaired community and the rest of society. Real-time Sign Language Recognition (SLR) is a state-of-the-art subject that promises to facilitate communication between the hearing-impaired community and others. Our research uses transfer learning to provide vision-based sign language recognition. We investigated recent works that use CNN-based methods and provided a literature review on deep learning systems for the sign language recognition (SLR) problem. This paper discusses the architecture of deep learning methods for SLR systems and explains a transfer learning application for fingerspelling sign classification. For the experiments, we used the Azerbaijani Sign Language Fingerspelling dataset and got 88.0% accuracy.

show abstract

A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation

Cited by 2 publications

References 30 publications

Frame-wise Action Representations for Long Videos via Sequence Contrastive Learning

Frame-wise Action Representations for Long Videos via Sequence Contrastive Learning

Transfer learning for Azerbaijani Sign Language Recognition

Contact Info

Product

Resources

About