Exploration of multimodal input interaction based on goals

Abstract

Today applications demand newer and newer ways through which humans can interact with the system for effective and natural interaction. Each interaction modality have unique features and supports different interaction goals, which makes them logically appropriate for specific types of interaction necessary for different applications. With several multimodal input technologies now being available for users, a challenge is to design optimal combinations of multiple modalities for specific tasks. Also, it is important to understand that, how these modalities may coherently be used together in a well-coordinated manner. This paper presents our initial research wherein we did exploration with the objective of designing a scheme for identification of optimal multimodal input interaction for various applications and use cases. We identified the key interaction goals of few sample applications or usage scenarios, and systematically compared those with the capabilities of the modalities, in order to identify optimal multimodal combination for those applications.

Older adults' perspectives on multimodal interaction with a conversational virtual coach

“…In fact, some previous studies explored the factors influencing the choice among different modalities such as speech, text, and gestures. Factors such as efficiency, accuracy, privacy and security (Himmelsbach et al, 2015), user characteristics (Dumas et al, 2009;Ghosh and Joshi, 2013;Schüssel et al, 2013;Schaffer et al, 2015), attitudes toward technology, and quality perceptions and personality factors (Wechsung, 2014) have been shown to impact modality selection. Jian et al (2013) demonstrated the potential of multimodal interaction (interacting with multiple modalities such as text and speech) for older adults.…”