SignFind: A Synchronized Sign Language and Chinese Character Teaching Game for Chinese Deaf Children Using Gesture Recognition

Nie, Jiangbo; Zhao, Yijun; Yao, Bing; Zheng, Xiaoqing; Chen, jiayan; Yu, Changchao; Hansen, Preben; Liu, Jianhui; Wang, Jiadi; Yan, Ge; Zhou, Lixiao; Ying, Fangtian; Liang, Xiubo; Wang, Guanyun

doi:10.1145/3491101.3519697

Cited by 5 publications

(3 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In spite of the ill effects of excessive gaming, digital games have immersive properties, demand active participation, challenges an individual to develop new skills, and provide emotional and social support, which has potential to bene it the learning experience [21,22]. Due to these reasons, games have been developed for a variety of sign languages across the world, including American SL [23,24], Australian SL [25], Arabic SL [26], Chinese SL [27], Brazilian SL [28], and Indian SL [29]. Based on the interaction mechanism, these sign language games can be divided into two broad categories: (1) learn-by-view: wherein the game shows signing videos/avatars to the player to help them learn new signs [26,30,25,28], and (2) learn-by-practice: wherein apart from the signing videos/avatars, the game prompts the player to mimic signs with feedback to help them improve the correctness of their signing [23,27,24,29].…”

Section: Sign Language Learning Gamesmentioning

confidence: 99%

“…Due to these reasons, games have been developed for a variety of sign languages across the world, including American SL [23,24], Australian SL [25], Arabic SL [26], Chinese SL [27], Brazilian SL [28], and Indian SL [29]. Based on the interaction mechanism, these sign language games can be divided into two broad categories: (1) learn-by-view: wherein the game shows signing videos/avatars to the player to help them learn new signs [26,30,25,28], and (2) learn-by-practice: wherein apart from the signing videos/avatars, the game prompts the player to mimic signs with feedback to help them improve the correctness of their signing [23,27,24,29]. Sign my World [25] is a learn-by-view Australian Sign Language game, to familiarize DHH children with commonly-used nouns and verb signs.…”

Section: Sign Language Learning Gamesmentioning

confidence: 99%

“…Although it combines gameplay with sign language learning, the research work lacks evaluation with end users. SignFind [27] is a recent work that requires the players to wear wrist sensors and sign speci ic words in Chinese Sign Language to explore a virtual world. Similar to CopyCat, SignFind uses a gesture recognition software running on a desktop/laptop.…”

Section: Sign Language Learning Gamesmentioning

confidence: 99%

See 2 more Smart Citations

SignIt! An Android game for sign bilingual play that collects labelled sign language data

Roshni Poddar,

Pradyumna YM,

Divya Prabha Jayakumar

et al. 2023

ITU J-FET

View full text Add to dashboard Cite

The Deaf or Hard-of-Hearing (DHH) community constitutes over 430 million people globally, with about 70 million of them using sign language as their primary means of communication. India has around 63 million DHH individuals. The DHH community in India faces several challenges, particularly in learning sign language and English, due to delayed diagnosis, stigma, oralism, and a diversity of languages. Digital games for spoken and sign language learning have gained popularity due to their advantages over traditional language learning methods, such as enhanced engagement and socialization, driving increased research and adoption over traditional methods. Moreover, the development of robust machine learning models for sign language recognition, which could significantly improve access for signers, is currently impeded by the scarcity of labelled sign language data. To address these challenges, we collaborated with NISH, an academic institution for the DHH community and developed SignIt!, an accessible and inclusive quiz platform that facilitates the learning of English and Indian Sign Language (ISL) with a secondary goal of data collection. To assess the game's usability, we conducted a study with 20 members of the DHH community, followed by interviewing 15 participants. Overall, our participants answered 2160 quiz questions and created 210 questions. The quiz creation resulted in the collection of three hours of labelled real-world sign language data. The interviews revealed novel insights, such as a preference for playing competitively with friends, empowerment by their agency to be content creators, and early signs of learning English, sign language, and quiz content by playing and creating quizzes. We plan to open-source and release SignIt! to increase its adoption among diverse DHH communities.

show abstract