A Hindi Virtual Keyboard Interface with Multimodal Feedback: A Case Study with a Dyslexic Child

Abstract

Up to 15% of the Indian school-going children suffer from dyslexia. This paper aims to determine the extent to which existing knowledge about the eye-tracking based human-computer interface can be used to assist these children in their reading and writing activities. A virtual keyboard system with multimodal feedback is proposed and designed for a lexically and structurally complex language and optimized for multimodal feedback involving several portable, non-invasive, and low-cost input devices: a touch screen, an eye-tracker, and a soft-switch. The performance was evaluated in terms of text-entry rate, information transfer rate, and type of errors with three different experimental conditions: 1) touch-screen condition with auditory feedback 2) eye-tracking condition with auditory and visual feedback, and 3) eye-tracking and soft-switch condition with auditory and visual feedback. The proposed multimodal feedback has shown a significant improvement in text-entry rate with less error. This work represents the first virtual keyboard with multimodal feedback for dyslexic children in the Hindi language, which can be extended to other languages.

Detection of Dyslexic Children Using Machine Learning and Multimodal Hindi Language Eye-Gaze-Assisted Learning System

“…In the ETSS modality, the eye-tracker was used for pointing to the item that can be then selected by pressing the soft-switch. HCI modalities include two visual (positive and negative feedback) and one auditory feedback (beep after successful execution of each command) [12]. In the gaze-based VK application, efficient feedback is necessary for the user so that the intended command box/character was selected to avoid errors in copy spelling and increase efficiency [22].…”

“…Dyslexic readers of the Hindi language have a complication in developing highquality, segmentally organized phonological representations of words and display poor blending skills [11]. While a pilot study suggested the possibility to enhance the awareness of a child with dyslexia through an adaptive multimodal interface using eye-tracking [12], this trend needs to be demonstrated rigorously with a collection of children with dyslexia and a control group, as proposed in this paper.…”

Design and evaluation of a time adaptive multimodal virtual keyboard

“…Other dwell-free techniques include multimodal and hybrid interfaces. These techniques address issues highlighted in previous studies [18,[42][43][44][45][46][47][48]. In particular, these studies have introduced a dwell-free technique for an eye-typing system, which focused on a combination of different modalities such as eye-tracking, smiling movements, input switches, and speech recognition.…”