Read and Attend: Temporal Localisation in Sign Language Videos

Abstract

The objective of this work is to annotate sign instances across a broad vocabulary in continuous sign language. We train a Transformer model to ingest a continuous signing stream and output a sequence of written tokens on a largescale collection of signing footage with weakly-aligned subtitles. We show that through this training it acquires the ability to attend to a large vocabulary of sign instances in the input sequence, enabling their localisation. Our contributions are as follows: (1) we demonstrate the ability to leverage large quantities of continuous signing videos with weakly-aligned subtitles to localise signs in continuous sign language; (2) we employ the learned attention to automatically generate hundreds of thousands of annotations for a large sign vocabulary; (3) we collect a set of 37K manually verified sign instances across a vocabulary of 950 sign classes to support our study of sign language recognition; (4) by training on the newly annotated data from our method, we outperform the prior state of the art on the BSL-1K sign language recognition benchmark.

Visual Keyword Spotting with Attention

“…In this section, we investigate the application of our method for spotting mouthed words in sign language videos. This is an important application of visual KWS, as it has enabled an entire line of work on sign language recognition [5,44,62]. Data description & evaluation protocol.…”

Aligning Subtitles in Sign Language Videos

“…Several studies have sought to employ subtitles as weak supervision for learning to localise and classify signs, using apriori mining [17] and multiple-instance learning [6,7,46]. More recent work has leveraged cues such as mouthings [2] and visual dictionaries [42] and by making use of deep neural network features with sliding window classifiers [37] and attention learned via a proxy translation task [56]. In deviation from these works, our objective is to localise complete subtitle units, rather than individual signs.…”

“…Video features. The visual features are 1024-dimensional embeddings extracted from the I3D [13] sign classification model made publicly available by the authors of [56]. The features are pre-extracted over sign language video segments.…”

“…Backbone pretraining. The I3D model is pretrained to perform 1064-way classification across the sign spotting instances with mouthings [2] and dictionary exemplars [42] (further details can be found in [56]). The model is then frozen and used to densely pre-extract visual features with stride 1 over the clips of the datasets.…”

“…Previous work exploiting such weakly-aligned data has mainly focused on finding sparse correspondences between keywords in the subtitle and individual signs [2,42,56], as opposed to localising the start and end times of a complete subtitle text in continuous signing. Though, as we show, localising isolated signs identified by keyword spotting nevertheless forms a useful pretraining task for full subtitle alignment.…”