Inclusive AR-games for Education of Deaf Children: Challenges and Opportunities

Abstract

Game-based learning has had a rapid development in the 21st century, attracting an increasing audience. However, inclusion of all is still not a reality in society, with accessibility for deaf and hard of hearing children as a remaining challenge. To be excluded from learning due to communication barriers can have severe consequences for further studies and work. Based on previous research Augmented Reality (AR) games can be joyful learning tools that include activities with different sign languages, but AR based learning games for deaf and hard of hearing lack research. This paper aims to present opportunities and challenges of designing inclusive AR games for education of deaf children. Methods involved conducting a scoping review of previous studies about AR for deaf people. Experts were involved as co-authors for in-depth understanding of sign languages and challenges for deaf people. A set of AR input and output techniques were analysed for appropriateness, and various AR based game mechanics were compared. Results indicate that inclusive AR gameplay for deaf people could be built on AR based image and object tracking, complemented with sign recognition. These technologies provide input from the user and the real-world environment typically via the camera to the app. Scene tracking and GPS can be used for location-based game mechanics. Output to the user can be done via local signed videos ideally, but also with images and animations. Moreover, a civic intelligence approach can be applied to overcome many of the challenges that have been identified in five dimensions for inclusion of deaf people i.e., cultural, educational, psycho-social, semantic, and multimodal. The input from trusted, educated signers and teachers can enable the connection between real world objects and signed videos to provide explanations of concepts. The conclusion is that the development of an inclusive, multi-language AR game for deaf people needs to be carried out as an international collaboration, addressing all five dimensions.

Mathematics and Sign Language Learning with a Tangible Game: An Inclusive Approach for DHH and Hearing Children

“…Accessibility in games relates to the design of the game being adequate to the needs of its players, whether they are related to vision, hearing, motor skills, cognition, or any other (Cezarotto et al, 2022). In the specific case of the Deaf community, creating an accessible and inclusive game implies the integration of sign language videos, witch should take in consideration five dimensions: (1) cultural, where the participation of someone from the community is key for the game to be welcomed in the community; (2) educational, it is necessary that the sign interpreter understands the concept of what is translating; (3) psychosocial, the DHH person must trust on what is being translated; (4) semantic, sign languages have different semantics in relation to the spoken language, as such it might be hard to find the corresponding signs to translate, when that happens the interpreter might resort to fingerspelling; and (5) multimodal, sign language is usually done with the combination of facial expressions and signs and has its one grammatical structure, as such it could be hard to have an accurate translation of the material if not by someone 'real' instead of virtual characters, since the accuracy of the hand movements, location, and orientation, as well as the use of facial expressions, are crucial for conveying the message (Westin et al, 2022).…”

ICTs in education for Deaf and Hard-Of-Hearing learners

“…The incorporation of written text and sign language translation into the game design are two potential options that may be helpful to overcome language and communication obstacles. They can both be experienced through an intact sensory modality, namely sight, and do not reliant on the audio modality, making them both accessible to DHH patients (Westin et al, 2022).…”