Multimodal Trajectory Playback for Teaching Shape Information and Trajectories to Visually Impaired Computer Users

Abstract

There are difficulties in presenting non-textual or dynamic information to blind or visually impaired users through computers. This paper examines the potential of haptic and auditory trajectory playback as a method of teaching shapes and gestures to visually impaired people. Two studies are described which test the success of teaching simple shapes. The first study examines haptic trajectory playback alone, played through a forcefeedback device and compares performance of visually impaired users with sighted users. It demonstrates that the task is significantly harder for visually impaired users. The second study builds on these results, combining force-feedback with audio to teach visually impaired users to recreate shapes. The results suggest that users performed significantly better when presented with multimodal haptic and audio playback of the shape rather than haptic only. Finally, an initial test of these ideas in an application context is described, with sighted participants describing drawings to visually impaired participants through touch and sound. This study demonstrates in what situations trajectory playback can provide a useful role in a collaborative setting.

Investigating touchscreen accessibility for people with visual impairments

“…For example, drawing arrows on the screen (one of the gestures proposed by Klok et al [13]) requires such awareness to be performed properly. Our participant also identified the issues discussed by both Crossan and Brewster, [15] and Klok et al [13], on how users would be trained on the gestures, especially if the device was consumer electronics, such as an MP3 player, which precludes individual training, and the user had to operate the device "out of the box".…”

“…Crossan and Brewster [15], have investigated teaching shape information using automatic playback with a PHANTOM Omni haptic device and dynamic audio feedback. In their system, two users (one sighted and one visually impaired) collaborated to draw a 2D shape.…”

“…Special force feedback technologies, such as by Crossan and Brewster [15] and Klok et al [13] are used, or some residual sight is assumed, such as with Leonard et al [17]. Many of these technologies and techniques may not be possible in the kinds of situations (such as using ticket machines) that we discuss here.…”

“…A way around this may be to employ technology to provide awareness of the user location. Crossan and Brewster [15] used dynamic audio to make participants aware of their location in the drawing frame, and this may be possible here.…”

A Model to Develop Videogames for Orientation and Mobility

“…Interfaces: In this stage of the model, the different interfaces that the blind user will use to develop the previously defined tasks are implemented. These interfaces can be of different sorts: Audio [14], haptic [8] and multimodal [4]. These interfaces must include the users' characteristics (from the Definition of HCI component), and the elements that make the orientation and mobility tasks possible.…”