Output Multimodal Interaction: The Case of Augmented Surgery

Abstract

Output multimodal interaction involves choice and combination of relevant interaction modalities to present information to the user. In this paper, we present a framework based on reusable software components for rapidly developing output multimodal interfaces by choosing and combining interaction modalities. Such an approach enables us to quickly explore several design alternatives as part of an iterative design process. Our approach is illustrated by examples from a computer-assisted surgery system that runs in a specific environment (i.e. an operating room) and so needs adapted multimodal interaction. Our approach supports the exploration of several output multimodal interaction design alternatives with the surgeons.

Cognitive-Aware Modality Allocation in Intelligent Multimodal Information Presentation

“…In existing IMMP studies, modality allocation is commonly rule-based [2,20,28,29,33,43,44,56,57,59]. Modality allocation rules typically associate factors with preferred modality choices.…”

“…• Application specific factor: when the needle is outside the patient's body, use only sound to present the distance to the target; when the needle is inserted into the body, use both sound and color gauge; when the needle tip is very near the target point (< 10 mm), use only color gauge (in [29] for surgery training).…”

“…The application domain of IMMP is very broad, including home entertainment [19], technical document generation [58], medical training [29], crisis management support [22], and much more. IMMP systems have been defined as knowledge-based systems, which exploit their knowledge base in order to dynamically adapt their design decisions to the run-time requirements of user-computer interaction, such as the user profile, task characteristics, nature of the information to be conveyed, etc.…”

Multimodal information presentation for high load human computer interaction

“…The development of IMMP has received much research attention during the past two decades. The application domain is very broad, including home entertainment [76], technical document generation [272], medical training [149], crisis management support [84] and much more. IMMP by definition have more than one modality available for generating presentations.…”

“…For example, in the computer-assisted surgery system presented in [149], the information to be presented was the distance between the needle tip and a target point inside the patient's body. An application-specific factor -the location of the needle -was selected to guide modality allocation.…”