SignIt! An Android game for sign bilingual play that collects labelled sign language data

Abstract

The Deaf or Hard-of-Hearing (DHH) community constitutes over 430 million people globally, with about 70 million of them using sign language as their primary means of communication. India has around 63 million DHH individuals. The DHH community in India faces several challenges, particularly in learning sign language and English, due to delayed diagnosis, stigma, oralism, and a diversity of languages. Digital games for spoken and sign language learning have gained popularity due to their advantages over traditional language learning methods, such as enhanced engagement and socialization, driving increased research and adoption over traditional methods. Moreover, the development of robust machine learning models for sign language recognition, which could significantly improve access for signers, is currently impeded by the scarcity of labelled sign language data. To address these challenges, we collaborated with NISH, an academic institution for the DHH community and developed SignIt!, an accessible and inclusive quiz platform that facilitates the learning of English and Indian Sign Language (ISL) with a secondary goal of data collection. To assess the game's usability, we conducted a study with 20 members of the DHH community, followed by interviewing 15 participants. Overall, our participants answered 2160 quiz questions and created 210 questions. The quiz creation resulted in the collection of three hours of labelled real-world sign language data. The interviews revealed novel insights, such as a preference for playing competitively with friends, empowerment by their agency to be content creators, and early signs of learning English, sign language, and quiz content by playing and creating quizzes. We plan to open-source and release SignIt! to increase its adoption among diverse DHH communities.