A 3D Descriptive Model for Designing Multimodal Feedbacks in any Virtual Environment for Gesture Learning

Abstract

This paper addresses the problem of creation and re-usability of pedagogical feedbacks, in Virtual Learning Environments (VLE), adapted to the needs of teachers for gesture learning. One of the main strengths of VLE is their ability to provide multimodal (i.e. visual, haptic, audio, etc.) feedbacks to help the learners in evaluating their skills, the task progress or its good execution. The feedback design strongly depends on the VLE and the pedagogical strategy. In addition, past studies mainly focus on the impact of the feedback modality on the learning situation, without considering others design elements (e.g. triggering rules, features of the motion to learn, etc.). However, most existing gesture-based VLEs are not editable without IT knowledge and therefore, failes in considering the evolution of pedagogical strategies. Consequently, this paper presents the GEstural FEedback EDitor (GEFEED) allowing non-IT teachers to create their multimodal and pedagogical feedbacks into any VLE developed under Unity3D. This editor operationalises a three dimensional descriptive model (i.e. feedback virtual representation, its triggering rules, involved 3D objects) of a pedagogical feedback dedicated to gesture leaning. Five types of feedbacks are proposed (i.e. visual color or text, audio from a file or a text and haptic vibration) and can be associated with four kinds of triggers (i.e. time, contact between objects, static spatial configuration, motion metric). In the context of a dilution task in biology, an experimental study is conducted in which teachers generate their feedbacks according to pre-defined or chosen pedagogical objectives. The results mainly show : (a) the acceptance of GEEFED and the underlying model and (b), the most used types of modalities (i.e. visual color, vibration, audio from text), triggering rules (i.e. motion metric, spatial configuration and contact) and (c), the teacher satisfaction in reaching their pedagogical objectives.