Digital Reading Support for The Blind by Multimodal Interaction

Abstract

Slate-type devices allow Individuals with Blindness or Severe Visual Impairment (IBSVI) to read in place with the touch of their fingertip by audio-rendering the words they touch. Such technologies are helpful for spatial cognition while reading. However, users have to move their fingers slowly or they may lose place on screen. Also, IBSVI may wander between lines without realizing they did. In this paper, we address these two interaction problems by introducing dynamic speech-touch interaction model, and intelligent reading support system. With this model, the speed of the speech will dynamically change coping up with the user's finger speed. The proposed model is composed of: 1-Audio Dynamics Model, and 2-Off-line Speech Synthesis Technique. The intelligent reading support system predicts the direction of reading, corrects the reading word if the user drifts, and notifies the user using a sonic gutter to help her from straying off the reading line. We tested the new audio dynamics model, the sonic gutter, and the reading support model in two user studies. The participants' feedback helped us fine-tune the parameters of the two models. Finally, we ran an evaluation study where the reading support system is compared to other VoiceOver technologies. The results showed preponderance to the reading support system with its audio dynamics and intelligent reading support components.

Conversational User Interfaces on Mobile Devices

“…Text entry [9,38,67,71,82,85,93] Application control [22,90,92] Speech analysis [26,36,52,68,72] Spoken output [28,31,66,77] Conversational Agents [33,43,45,48,55,61,96] Probes [18,74] Domains Education [22,43,52,61,68,77] Health & Well-being [26,28,36,71,72,92,94] Transportation [33,38,55,67,90] Gaming [18,26,36,52,66,…”

“…A number of systems focused only on generating spoken output for specific purposes. El-Glaly [28] developed an intelligent reading support system for blind people. It harnessed spatial, auditory, and haptic feedback to follow the user's finger across text, which it would read to them.…”

Rule based intelligent system verbalizing mathematical notation

“…Different approaches have been used for automatic conversion to/from spoken form of mathematical notation. One possible solution for the system to read (or write) mathematics in an understandable and proper way is to equip it with a tool for machine translation from mathematical symbolic notation to spoken language [13] and the speech synthesis [18] which often needs additional efforts for non-English national languages [36]. Most of the existing TTS systems intended for mathematics are focused on particular type of input (like L A T E X or MathML) and use a set of pre-defined translation rules in the translation process.…”

Read

“…Several initiatives emerged to provide non-linear access to content, such as concurrent speech [5] or aural glancing of texts [4]. Multimodality is also pointed as a solution to improve content accessibility regarding the potential information overload raised by linear access to information [3]. This kind of approach is also applied for accessing complex content such as mathematics [1].…”