Empirical Evaluation of Multimodal Input Interactions

Abstract

With variety of interaction technologies like speech, pen, touch, hand or body gestures, eye gaze, etc., being now available for users, it is a challenge to design optimal and effective multimodal combinations for specific tasks. For designing that, it is important to understand how these modalities can be combined and used in a coordinated manner. We performed an experimental evaluation of combinations of different multimodal inputs, such as keyboard, speech and touch with pen etc, in an attempt to investigate, which combinations are efficient for diverse needs of the users. In our study, multimodal combination of three modalities was found to be more effective in terms of performance, accuracy and user experience than that of two modalities. Further, we also inferred the roles that each of the modalities play in a multimodal combination to achieve the usability goals.

Exploration of multimodal input interaction based on goals

“…Instead of being generic as in the prior related works, we approached the problem by considering the diverse application types and several interaction technologies. One of our earlier works [14] presented the evaluation of different multimodal input combinations for typical desktop tasks, in terms of performance, accuracy and user experience. The said work provides evaluation from the interaction modalities perspective, or the systems perspective (modality-centered).…”

“…Similar to the earlier section on application goals, this section presents the survey results on interaction modality goals by bringing in the real world examples and reasoning. In one of our earlier works [14], we had performed experimental evaluation of three interaction modalities, keyboard, speech and touch with pen, where these modalities were evaluated based on three attributes, Performance, Accuracy and Experience. Our analysis showed that, in terms of Performance, Speech and Touch performed better than Keyboard, while Keyboard was found to be the most accurate, and Speech to be the least accurate among those three modalities.…”

Multimodal Approaches for Text Entry in Indian Language on Mobile Devices

“…On a parallel track we performed few controlled experiments wherein the users were asked to perform few tasks such as navigation and text editing by making simultaneous use of multiple interaction modalities. We evaluated their level of performance, accuracy and user experience [8] as well as analysed few of their usage patterns such as patterns of errors, modality switching and modality preferences [9]. We analysed the steps required to input various types of characters namely, consonants, matras (vowel marks), conjuncts, diacritic marks for four popular Indic language keyboards.…”