A Dataset for Linguistic Understanding, Visual Evaluation, and Recognition of Sign Languages: The K-RSL

Abstract

The paper presents the first dataset that aims to serve interdisciplinary purposes for the utility of computer vision community and sign language linguistics. To date, a majority of Sign Language Recognition (SLR) approaches focus on recognising sign language as a manual gesture recognition problem. However, signers use other articulators: facial expressions, head and body position and movement to convey linguistic information. Given the important role of non-manual markers, this paper proposes a dataset and presents a use case to stress the importance of including non-manual features to improve the recognition accuracy of signs. To the best of our knowledge no prior publicly available dataset exists that explicitly focuses on non-manual components responsible for the grammar of sign languages. To this end, the proposed dataset contains 28250 videos of signs of high resolution and quality, with annotation of manual and nonmanual components. We conducted a series of evaluations in order to investigate whether non-manual components would improve signs' recognition accuracy. We release the dataset to encourage SLR researchers and help advance current progress in this area toward realtime sign language interpretation. Our dataset will be made publicly available at https:// krslproject.github.io/krsl-corpus

A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation

“…Sign Language Recognition. A fundamental task in sign language understanding is Isolated Sign Language Recognition (ISLR), which aims to identify a single gloss word label for a short video clip [2,19,20,28,30,43]. The more challenging task of Continuous Sign Language Recognition (CSLR) seeks to convert a continuous sign video into a gloss sequence using only weak sentence-level annotations [9,22,23,37,50].…”