Towards a taxonomy of error-handling strategies in recognition-based multi-modal human–computer interfaces

Abstract

In this paper, we survey the different types of error-handling strategies that have been described in the literature on recognition-based human-computer interfaces. A wide range of strategies can be found in spoken human-machine dialogues, handwriting systems, and multimodal natural interfaces.We then propose a taxonomy for classifying error-handling strategies that has the following three dimensions: the main actor in the error-handling process (machine versus user), the purpose of the strategy (error prevention, discovery, or correction), and the use of different modalities of interaction. The requirements that different error-handling strategies have on different sets of interaction modalities are also discussed. The main aim of this work is to establish a classification that can serve as a tool for understanding how to develop more efficient and more robust multimodal human-machine interfaces.Keywords: recognition-based technology, multimodal interfaces, error-handling, taxonomy, interaction design, interaction robustness Recognition-based technologyMultimodal interaction refers to interaction with the virtual and physical environment through natural modes of communication such as speech, body gestures, handwriting, graphics, or gaze.Unlike keyboards and mice inputs, natural modes of communication usually are non-deterministic, and have to be "recognised" by a recognition system, before they can be passed on to an application.Recent developments in recognition-based technology (e.g. speech and gesture recognition) have opened a myriad of new possibilities for the design and implementation of multimodal applications.Handwriting recognisers, for example, are being used in personal digital assistants (e.g. Paragon's multilingual PenReader software for Pocket PC devices), and speech recognition has made its way into desktop machines (e.g. IBM's ViaVoice TM speech recognition engines). However, designing and

Development and Evaluation of Automotive Speech Interfaces: Useful Information from the Human Factors and the Related Literature

“…Strategies to correct errors include repeating and rephrasing the utterances, spelling out words, contradicting a system response, correcting using a different modality (e.g., manual entry instead of speech), and restarting, among others [59][60][61][62].…”

Understanding Gesture and Speech Multimodal Interactions for Manipulation Tasks in Augmented Reality Using Unconstrained Elicitation

“…They might say "roll clockwise" and perform a counterclockwise movement with their hand. Multimodal systems can suffer from compounding errors caused by incorrect recognition, or mismatched interactions such as the ones seen in this study [6]. These errors could take more time than standard uni-modal errors to correct or cause compounding errors when a second error is made during an attempt to correct the first.…”

Fishing or a Z?

“…Human factors research in multimodal interaction concerned with recognition errors [11] is a well researched topic in multimodal interfaces [2,13,15], where investigations were typically concerned with error handling strategies devised by users in the face of recognition errors (e.g., modality switching to a 'familiar', more efficient modality). In speech-based interfaces, a common finding is that the most intuitive and instinctive way for correcting errors in speech is to repeat the spoken utterance [21] and hyperarticulate it [14].…”

“…In a follow-up study by [6], they tested 3 commercial Automatic Speech Recognition (ASR) systems where they found that a little over 50% of the time, subjects would continue to repeat the utterance to a spiral depth of level 3. However, while recognition errors have been well studied in domains such as speech-based interfaces [2,14], handwriting recognition [16,17], and multimodal interfaces [13], less attention has been given to usability issues surrounding device-based gesture interaction. An exception is the study by [8], where they investigated user tolerance for errors in touch-less computer vision-enabled gesture interaction under both desktop (keyboard readily available in front of subjects) and ubiquitous computing settings (keyboard not readily available).…”