CopyCat: Using Sign Language Recognition to Help Deaf Children Acquire Language Skills

Abstract

Deaf children born to hearing parents lack continuous access to language, leading to weaker working memory compared to hearing children and deaf children born to Deaf parents. CopyCat is a game where children communicate with the computer via American Sign Language (ASL), and it has been shown to improve language skills and working memory. Previously, CopyCat depended on unscalable hardware such as custom gloves for sign verifcation, but modern 4K cameras and pose estimators present new opportunities. Before re-creating the CopyCat game for deaf children using of-the-shelf hardware, we evaluate whether current ASL recognition is sufcient.Using Hidden Markov Models (HMMs), user independent word accuracies were 90.6%, 90.5%, and 90.4% for AlphaPose, Kinect, and MediaPipe, respectively. Transformers, a state-of-the-art model in natural language processing, performed 17.0% worse on average.Given these results, we believe our current HMM-based recognizer can be successfully adapted to verify children's signing while playing CopyCat. CCS Concepts: • Human-centered computing → Ubiquitous and mobile computing systems and tools; Accessibility technologies; • Applied computing → Computer-managed instruction; • Computing methodologies → Machine learning; Feature selection.

SignIt! An Android game for sign bilingual play that collects labelled sign language data

“…Virtual Sign [31] is a similar game that uses sensor gloves and Microsoft Kinect for learning Portuguese Sign Language. Although learn-by-practice games are better suited for learning, they require expensive hardware like gloves with embedded accelerometers [23], pinch gloves [24] and depth cameras [32] for tracking hand gestures, thus limiting their widespread adoption. Moreover, the desktop/laptop requirement restricts the gameplay to a non-mobile setting.…”