The Rensselaer Mandarin Project — A Cognitive and Immersive Language Learning Environment

Abstract

The Rensselaer Mandarin Project enables a group of foreign language students to improve functional understanding, pronunciation and vocabulary in Mandarin Chinese through authentic speaking situations in a virtual visit to China. Students use speech, gestures, and combinations thereof to navigate an immersive, mixed reality, stylized realism game experience through interaction with AI agents, immersive technologies, and game mechanics. The environment was developed in a black box theater equipped with a human-scale 360◦ panoramic screen (140h, 200r), arrays of markerless motion tracking sensors, and speakers for spatial audio.

Exploring Kid Space in the wild: a preliminary study of multimodal and immersive collaborative play-based learning experiences

“…Similarly, Rensselaer Mandarin is another recent research project enabling students to use speech and gestures to interact with an immersive, mixed reality game experience. Students interact with artificial intelligence (AI) agents through a human-scale, 360° panoramic screen to improve understanding, pronunciation, and vocabulary in Mandarin Chinese (Allen et al, 2019 ).…”

Same benefits, different communication patterns: Comparing Children's reading with a conversational agent vs. a human partner

“…Some other projects have developed intelligent agents embodied in avatars. Allen and colleagues utilized an avatar agent to speak with students in authentic situations, with a goal of improving students' comprehension, pronunciation, and vocabulary in a foreign language (Allen, Divekar, Drozdal, Balagyozyan, Zheng, Song et al, 2019). Authors incorporated an agent in a children's science animation series to teach children scientific vocabularies, and this agent was embodied in the series' main character (Xu & Warschauer, 2020d).…”

The Development of Pedagogical Functions in Entertainment Contents

“…According to the report, after the completion of the project, the experience feedback is overall positive, and the expected effect has been achieved [17]. Under the premise of assuring the effect of native-style immersive learning of language, the project solves the mobility barrier of cross-border entry into a new language environment to achieve expected learning results.…”

“…The program aims to simulate an immersive Chinese daily living environment by employing VR technology and an interactive game engine. According to the project report, this "teaching process" is more precisely a self-motivated gaming experience, which happens in a black-box theatre instead of an ordinary classroom [17]. The learning experience is based on a game-like exploration supported by motion sensors and spatial-audio speakers, which turns the physical space into a CIR (cognitive immersive room) space.…”