Multimodal Fusion and Fission within the W3C MMI Architectural Pattern

Schnelle-Walka, Dirk; Duarte, Carlos; Radomski, Stefan

doi:10.1007/978-3-319-42816-1_19

Cited by 2 publications

(3 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Modality fusion has attained maturity [18], but fusion engines are not yet integrated in MMI architectures. Our team recently presented a first proposal regarding how to integrate a fusion engine in a W3C-aligned MMI architecture [19], but to tackle fusion in the generic decoupled scenario of an MMI architecture, where a wide range of modalities can exist and, potentially, be fused, several challenges need to be addressed [20]. For instance, the semantics for the input modalities must be uniform: saying “left” or pressing the left arrow should result in the same semantic content, e.g., LEFT.…”

Section: Background and Related Workmentioning

confidence: 99%

“…The adaptive, context-aware use of output modalities—fission—while it has deserved far less attention than fusion, can play a very important role in how interactive systems can adapt to different devices, users and dynamic contexts [21]. In this regard, very interesting contributions have been made by [22,23], dividing the life cycle of a multimodal presentation in stages, as also proposed by the W3C (generation, styling, and rendering [13,20,24]), proposing a mathematical model to determine the best possible combination of input and output modalities based on the user’s profile and preferences; and our group proposing the basis for an intelligent adaptation of output relying on context and user models—AdaptO [6].…”

Section: Background and Related Workmentioning

confidence: 99%

See 1 more Smart Citation

The AM4I Architecture and Framework for Multimodal Interaction and Its Application to Smart Environments

Almeida

Teixeira

Silva

et al. 2019

Sensors

View full text Add to dashboard Cite

Technologies, such as smart sensors, actuators, and other kinds of devices, are often installed in our environments (e.g., our Homes) and available to integrate our daily lives. Despite their installation being motivated by the pursuit of automation and increased efficiency, making these environments usable, acceptable and enjoyable in a sustainable, energy efficient way is not only a matter of automation. Tackling these goals is a complex task demanding the combination of different perspectives including building and urban Architecture, Ubiquitous Computing and Human-Computer Interaction (HCI) to provide occupants with the means to shape these environments to their needs. Interaction is of paramount relevance in the creation of adequate relations of users with their environments, but it cannot be seen independently from the ubiquitous sensing and computing or the environment’s architecture. In this regard, there are several challenges to HCI, particularly in how to integrate this multidisciplinary effort. Although there are several solutions to address some of these challenges, the complexity and dynamic nature of the smart environments and the diversity of technologies involved still present many challenges, particularly for its development. In general, the development is complex, and it is hard to create a dynamic environment providing versatile and adaptive forms of interaction. To participate in the multidisciplinary effort, the development of interaction must be supported by tools capable of facilitating co-design by multidisciplinary teams. In this article, we address the development of interaction for complex smart environments and propose the AM4I architecture and framework, a novel modular approach to design and develop adaptive multiplatform multilingual multi-device multimodal interactive systems. The potential of the framework is demonstrated by proof-of-concept applications in two different smart environment contexts, non-residential buildings and smart homes.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Section: Background and Related Workmentioning

confidence: 99%

The AM4I Architecture and Framework for Multimodal Interaction and Its Application to Smart Environments

Almeida

Teixeira

Silva

et al. 2019

Sensors

View full text Add to dashboard Cite

show abstract

“…The main difference is the time at which level of processing the fusion of different sources takes place (Dumas et al, 2009). The following paragraphs give a brief summary based on (Dumas et al, 2009;Schnelle-Walka, Duarte, & Radomski, 2016):…”

Section: Fusion Methodsmentioning

confidence: 99%

Natural Multimodal Interaction in the Car - Generating Design Support for Speech, Gesture, and Gaze Interaction while Driving

Roider¹

View full text Add to dashboard Cite

Driving a modern car is more than just maneuvering the vehicle on the road. At the same time, drivers want to listen to music, operate the navigation system, compose, and read messages and more. Future cars are turning from simple means for transportation into smart devices on wheels. This trend will continue in the next years together with the advent of automated vehicles. However, technical challenges, legal regulations, and high costs slow down the penetration of automated vehicles. For this reason, a great majority of people will still be driving manually at least for the next decade. Consequently, it must be ensured that all the features of novel infotainment systems can be used easily, efficiently without distracting the driver from the task of driving and still provide a high user experience. A promising approach to cope with this challenge is multimodal in-car interaction. Multimodal interaction basically describes the combination of different input and output modalities for driver-vehicle interaction. Research has pointed out the potential to create a more flexible, efficient, and robust interaction. In addition to that, the integration of natural interaction modalities such as speech, gestures and gaze, the communication with the car could increase the naturalness of the interaction. Based on these advantages, the researcher community in the field of automotive user interfaces has produced several interesting concepts for multimodal interaction in vehicles. The problem is that the resulting insights and recommendations are often easily applicable in the design process of other concepts because they too concrete or very abstract. At the same time, concepts focus on different aspects. Some aim to reduce distraction while others want to increase efficiency or provide a better user experience. This makes it difficult to give overarching recommendations on how to combine natural input modalities while driving. As a consequence, interaction designers of in-vehicle systems are lacking adequate design support that enables them to transfer existing knowledge about the design of multimodal in-vehicle applications to their own concepts. This thesis addresses this gap by providing empirically validated design support for multimodal in-vehicle applications. It starts with a review of existing design support for automotive and multimodal applications. Based on that we report a series of user experiments that investigate various aspects of multimodal in-vehicle interaction with more than 200 participants in lab setups and driving simulators. During these experiments, we assessed the potentials of multimodality while driving, explored how user interfaces can support speech and gestures, and evaluated novel interaction techniques. The insights from these experiments extend existing knowledge from literature in order to create the first pattern collection for multimodal natural in-vehicle interaction. The collection contains 15 patterns that describe solutions for reoccurring problems when combining natural input with speech, gestures, or gaze in the car in a structured way. Finally, we present a prototype of an in-vehicle information system, which demonstrates the application of the proposed patterns and evaluate it in a driving-simulator experiment. This work contributes to field of automotive user interfaces in three ways. First, it presents the first pattern collection for multimodal natural in-vehicle interaction. Second, it illustrates and evaluates interaction techniques that combine speech and gestures with gaze input. Third, it provides empirical results of a series of user experiments that show the effects of multimodal natural interaction on different factors such as driving performance, glance behavior, interaction efficiency, and user experience.

show abstract

Multimodal Fusion and Fission within the W3C MMI Architectural Pattern

Cited by 2 publications

References 18 publications

The AM4I Architecture and Framework for Multimodal Interaction and Its Application to Smart Environments

The AM4I Architecture and Framework for Multimodal Interaction and Its Application to Smart Environments

Natural Multimodal Interaction in the Car - Generating Design Support for Speech, Gesture, and Gaze Interaction while Driving

Contact Info

Product

Resources

About