Multimodal 'eyes-free' interaction techniques for wearable devices

Abstract

Mobile and wearable computers present input/output problems due to limited screen space and interaction techniques. When mobile, users typically focus their visual attention on navigating their environment -making visually demanding interface designs hard to operate. This paper presents two multimodal interaction techniques designed to overcome these problems and allow truly mobile, 'eyes-free' device use. The first is a 3D audio radial pie menu that uses head gestures for selecting items. An evaluation of a range of different audio designs showed that egocentric sounds reduced task completion time, perceived annoyance, and allowed users to walk closer to their preferred walking speed. The second is a sonically enhanced 2D gesture recognition system for use on a belt-mounted PDA. An evaluation of the system with and without audio feedback showed users' gestures were more accurate when dynamically guided by audio-feedback. These novel interaction techniques demonstrate effective alternatives to visual-centric interface designs on mobile devices.

A user study of auditory versus visual interfaces for use while driving

“…By turning a knob and giving voice commands the user can interact with the system. Brewster et al (2003) created a mobile system based on Audio Windows by Cohen and Ludwig (1991). They used spatialized auditory icons localized in the horizontal plain either around or in front of the user's head.…”

AudioBrowser: a mobile browsable information access for the visually impaired

“…Brewster et al developed an eyes-free gesture-audio interface for wearable devices [8]. The interface receives input from the user's head movements and hand gestures, and outputs 3D audio through a headphone.…”

“…First, the prevalent use of gestures in human communications makes intuitive gesture input on computers possible. Gesture input commands can be designed so that traditional gesture meanings in human communications are carried on (e.g., [8]). Second, this type of input can provide advantages similar to those of direct manipulation on GUIs, and can reduce the memory demand for recalling commands (e.g., [15,31]).…”

Nomadic Speech-Based Text Entry: A Decision Model Strategy for Improved Speech to Text Processing

“…Consistent with the underlying motivation for the current research, it has been observed that mobile devices must tolerate a broad range of external conditions and technology must be adaptable to changing external conditions (Satyanarayanan, 1996). Several studies have explicitly examined interaction solutions for mobile users such as systems that support eyes-free interactions by avoiding the standard graphical user interface found on most desktop systems (Brewster, Lumsden, Bell, Hall, & Tasker, 2003), a minimal attention user interface designed specifically for field workers (Pascoe, Ryan, & Morse, 2000), and speech and audio-based interactions with mobile devices (Sawhney & Schmandt, 2000). Of importance, mobile data entry and multiple demands for visual attention are often mentioned as key challenges that must be addressed as researchers seek to develop new solutions, which will allow for the more effective use of highly portable computing devices.…”