SiAM - Situation-Adaptive Multimodal Interaction for Innovative Mobility Concepts of the Future

Mitrevska, Monika; Moniri, Mohammad Mehdi; Neßelrath, Robert; Schwartz, Tim; Feld, Michael S.; Korber, Yannick; Deru, Matthieu; Müller, Christian

doi:10.1109/ie.2015.39

Cited by 6 publications

(5 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As research has shown, the use of multiple input modalities can surpass a single input modality in terms of performance [9,18,37], multimodal user interaction offers a significant utility for in-vehicle application. Mitrevska et al demonstrate an adaptive control of in-vehicle functions using an individual modality (speech, gaze or gesture) or a combination of two or more [23]. Mu ller and Weinberg discuss methods for a multimodal interaction using gaze, touch and speech for in-vehicle tasks presenting a few advantages and disadvantages of individual modalities [26].…”

Section: Related Workmentioning

confidence: 99%

Multimodal Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the Vehicle

Aftab

Beeck

2022

27th International Conference on Intelligent User Interfaces

View full text Add to dashboard Cite

show abstract

Section: Related Workmentioning

confidence: 99%

Multimodal Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the Vehicle

Aftab

Beeck

2022

27th International Conference on Intelligent User Interfaces

View full text Add to dashboard Cite

show abstract

“…Multimodal user interaction has a wide variety of applications for in-vehicle functions. Mitrevska et al demonstrate the use cases of an adaptive multimodal control of in-vehicle functions with the help of an individual modality (speech, gaze or gesture) or a combination of two or more modalities [15].…”

Section: Related Workmentioning

confidence: 99%

You Have a Point There: Object Selection Inside an Automobile Using Gaze, Head Pose and Finger Pointing

Aftab,

von der Beeck,

Feld

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Sophisticated user interaction in the automotive industry is a fast emerging topic. Mid-air gestures and speech already have numerous applications for driver-car interaction. Additionally, multimodal approaches are being developed to leverage the use of multiple sensors for added advantages. In this paper, we propose a fast and practical multimodal fusion method based on machine learning for the selection of various control modules in an automotive vehicle. The modalities taken into account are gaze, head pose and finger pointing gesture. Speech is used only as a trigger for fusion. Single modality has previously been used numerous times for recognition of the user's pointing direction. We, however, demonstrate how multiple inputs can be fused together to enhance the recognition performance. Furthermore, we compare different deep neural network architectures against conventional Machine Learning methods, namely Support Vector Regression and Random Forests, and show the enhancements in the pointing direction accuracy using deep learning. The results suggest a great potential for the use of multimodal inputs that can be applied to more use cases in the vehicle. CCS CONCEPTS• Human-centered computing → HCI design and evaluation methods; • Computing methodologies → Neural networks; Support vector machines; Classification and regression trees.

show abstract

“…Participants of the evaluation study were much more comfortable speaking than nodding and were much more forgiving for recognition errors with the speech recognition than for than for head nods (Kousidis et al, 2014). C3 also enables the driver to choose between equivalent input modalities (Mitrevska et al, 2015). At the same time, the concept also uses complementary input of different modalities to build up a command.…”

Section: Providing Equivalent Alternativesmentioning

confidence: 99%

“…C8 integrates gaze information for controlling multiple Figure 2.12: Left: C5 uses complementary input of speech with touch gestures on the steering wheel (Pfleging et al, 2011(Pfleging et al, , 2012. Right: Sensor setup for combining gaze input with gestures on the steering wheel in C3 (Mitrevska et al, 2015).…”

Section: Complementary Inputmentioning

confidence: 99%

“…Researchers demonstrated the use of pointing towards targets inside (Ahmad et al, 2016), as well as outside of the car (Rümelin et al, 2013) as a feasible approach. The SIAM prototype uses gestures on the steering wheel to provide a reference to front windows, or outside mirrors and combines them with speech input to determine the function (Mitrevska et al, 2015).…”

Section: Examplesmentioning

confidence: 99%

See 1 more Smart Citation

Natural Multimodal Interaction in the Car - Generating Design Support for Speech, Gesture, and Gaze Interaction while Driving

Roider¹

View full text Add to dashboard Cite

Driving a modern car is more than just maneuvering the vehicle on the road. At the same time, drivers want to listen to music, operate the navigation system, compose, and read messages and more. Future cars are turning from simple means for transportation into smart devices on wheels. This trend will continue in the next years together with the advent of automated vehicles. However, technical challenges, legal regulations, and high costs slow down the penetration of automated vehicles. For this reason, a great majority of people will still be driving manually at least for the next decade. Consequently, it must be ensured that all the features of novel infotainment systems can be used easily, efficiently without distracting the driver from the task of driving and still provide a high user experience. A promising approach to cope with this challenge is multimodal in-car interaction. Multimodal interaction basically describes the combination of different input and output modalities for driver-vehicle interaction. Research has pointed out the potential to create a more flexible, efficient, and robust interaction. In addition to that, the integration of natural interaction modalities such as speech, gestures and gaze, the communication with the car could increase the naturalness of the interaction. Based on these advantages, the researcher community in the field of automotive user interfaces has produced several interesting concepts for multimodal interaction in vehicles. The problem is that the resulting insights and recommendations are often easily applicable in the design process of other concepts because they too concrete or very abstract. At the same time, concepts focus on different aspects. Some aim to reduce distraction while others want to increase efficiency or provide a better user experience. This makes it difficult to give overarching recommendations on how to combine natural input modalities while driving. As a consequence, interaction designers of in-vehicle systems are lacking adequate design support that enables them to transfer existing knowledge about the design of multimodal in-vehicle applications to their own concepts. This thesis addresses this gap by providing empirically validated design support for multimodal in-vehicle applications. It starts with a review of existing design support for automotive and multimodal applications. Based on that we report a series of user experiments that investigate various aspects of multimodal in-vehicle interaction with more than 200 participants in lab setups and driving simulators. During these experiments, we assessed the potentials of multimodality while driving, explored how user interfaces can support speech and gestures, and evaluated novel interaction techniques. The insights from these experiments extend existing knowledge from literature in order to create the first pattern collection for multimodal natural in-vehicle interaction. The collection contains 15 patterns that describe solutions for reoccurring problems when combining natural input with speech, gestures, or gaze in the car in a structured way. Finally, we present a prototype of an in-vehicle information system, which demonstrates the application of the proposed patterns and evaluate it in a driving-simulator experiment. This work contributes to field of automotive user interfaces in three ways. First, it presents the first pattern collection for multimodal natural in-vehicle interaction. Second, it illustrates and evaluates interaction techniques that combine speech and gestures with gaze input. Third, it provides empirical results of a series of user experiments that show the effects of multimodal natural interaction on different factors such as driving performance, glance behavior, interaction efficiency, and user experience.

show abstract

SiAM - Situation-Adaptive Multimodal Interaction for Innovative Mobility Concepts of the Future

Cited by 6 publications

References 3 publications

Multimodal Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the Vehicle

Multimodal Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the Vehicle

You Have a Point There: Object Selection Inside an Automobile Using Gaze, Head Pose and Finger Pointing

Natural Multimodal Interaction in the Car - Generating Design Support for Speech, Gesture, and Gaze Interaction while Driving

Contact Info

Product

Resources

About