American Sign Language Fingerspelling Recognition in the Wild

Abstract

We address the problem of American Sign Language fingerspelling recognition "in the wild", using videos collected from websites. We introduce the largest data set available so far for the problem of fingerspelling recognition, and the first using naturally occurring video data. Using this data set, we present the first attempt to recognize fingerspelling sequences in this challenging setting. Unlike prior work, our video data is extremely challenging due to low frame rates and visual variability. To tackle the visual challenges, we train a special-purpose signing hand detector using a small subset of our data. Given the hand detector output, a sequence model decodes the hypothesized fingerspelled letter sequence. For the sequence model, we explore attention-based recurrent encoder-decoders and CTC-based approaches. As the first attempt at fingerspelling recognition in the wild, this work is intended to serve as a baseline for future work on sign language recognition in realistic conditions. We find that, as expected, letter error rates are much higher than in previous work on more controlled data, and we analyze the sources of error and effects of model variants.Index Terms-American Sign Language, fingerspelling, connectionist temporal classification, attention models 2 Two-handed fingerspelling occasionally occurs, including in our data.

Fingerspelling Recognition in the Wild With Iterative Visual Attention

“…We make two main contributions: (1) We propose iterative attention, an approach for obtaining regions of interest of high resolution with limited computation (see Figure 3). Our model trained with iterative attention achieves higher accuracy than the previous best approach [35], which requires a custom hand detector. We further show that even when a hand or face detector is available, our approach provides significant added value.…”

“…In this paper we consider a constrained task (fingerspelling recognition), but with looser visual and stylistic constraints than in most previous work. The recently introduced Chicago Fingerspelling in the Wild (ChicagoFSWild) data set [35] consists of 7304 fingerspelling sequences from online videos. This data set includes a large number of signers (168) and a wide variety of challenging visual conditions, and we use it as one of our test beds.…”

“…We use two data sets: Chicago Fingerspelling in the Wild (ChicagoFSWild) [35], which was carefully annotated by experts; and a crowdsourced data set we introduce here, ChicagoFSWild+. Both contain clips of fingerspelling sequences excised from sign language video "in the wild", collected from online sources such as YouTube and deafvideo.tv.…”

“…Leftmost: example frame from the ChicagoFSVid studio data set [17]. Rest: Example frames from the ChicagoFSWild data set [35] (see Section 4). Figure 2: The ASL fingerspelling alphabet, from [16].…”

“…Hand detection is a typical first step, as sign language often involves long sequences of large image frames. For example, in a recently introduced fingerspelling data set [35], a substantial proportion of sequences are more than 100 frames long, with an average frame size of 720 × 480, while the informative region is on average only about 10%. An end-to-end recognition model on raw image frames may have prohibitive memory requirements.…”