Automatic Dense Annotation of Large-Vocabulary Sign Language Videos

Momeni, Liliane; Bull, Hannah; Prajwal, K R; Albanie, Samuel; Varol, Gül; Zisserman, Andrew

doi:10.1007/978-3-031-19833-5_39

Cited by 9 publications

(2 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, this work stands out due to its two unique aspects: (i) it has been shown to be data efficient, (ii) it contains a strong visual backbone that can attend to and extract accurate lip features. These visual features have been shown to work well in several other tasks such as visual keyword spotting [29] and even spotting mouth movements in sign language [24]. The sub-word units considered to learn the text representations help to model the ambiguities of the task significantly better than using characters.…”

Section: Our Approachmentioning

confidence: 99%

Towards Accurate Lip-to-Speech Synthesis in-the-Wild

Hegde,

Mukhopadhyay,

Jawahar

et al. 2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

Figure 1: We propose a novel approach for multi-speaker lip-to-speech synthesis in the wild. Prior works try to learn a language model directly from raw speech, which only provides weak supervision due to the presence of other acoustic variations such as voice, accents, and prosody. We solve this problem by relying on recent advancements in lip-to-text generation. We condition on the noisy text outputs and lip video to generate natural speech with clearly pronounced words.

show abstract

Section: Our Approachmentioning

confidence: 99%

Towards Accurate Lip-to-Speech Synthesis in-the-Wild

Hegde,

Mukhopadhyay,

Jawahar

et al. 2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…Sign language recognition has received much attention in recent years (Koller et al 2018;Momeni et al 2022;Niu and Mak 2020;Jin and Zhao 2021;Hu, Zhou, and Li 2021;Li et al 2020b). Typically, the research works can be grouped into two categories based on the input modality, i.e., RGBbased methods and pose-based methods.…”

Section: Sign Language Recognitionmentioning

confidence: 99%

BEST: BERT Pre-training for Sign Language Recognition with Coupling Tokenization

Zhao

Zhou

et al. 2023

AAAI

View full text Add to dashboard Cite

In this work, we are dedicated to leveraging the BERT pre-training success and modeling the domain-specific statistics to fertilize the sign language recognition~(SLR) model. Considering the dominance of hand and body in sign language expression, we organize them as pose triplet units and feed them into the Transformer backbone in a frame-wise manner. Pre-training is performed via reconstructing the masked triplet unit from the corrupted input sequence, which learns the hierarchical correlation context cues among internal and external triplet units. Notably, different from the highly semantic word token in BERT, the pose unit is a low-level signal originally locating in continuous space, which prevents the direct adoption of the BERT cross entropy objective. To this end, we bridge this semantic gap via coupling tokenization of the triplet unit. It adaptively extracts the discrete pseudo label from the pose triplet unit, which represents the semantic gesture / body state. After pre-training, we fine-tune the pre-trained encoder on the downstream SLR task, jointly with the newly added task-specific layer. Extensive experiments are conducted to validate the effectiveness of our proposed method, achieving new state-of-the-art performance on all four benchmarks with a notable gain.

show abstract