Audio-visual and multimodal speech-based systems

Gibbon, Dafydd; Mertins, Inge; Moore, Roger K.

doi:10.1007/978-1-4615-4501-9_2

Cited by 12 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[2] The significance of multimodal discourse analysis is that it can integrate language with other relevant meaning resources. [3] It can not only see the role played by the language system in the process of meaning exchange, but also see the effects of other symbol systems such as images, music, colors, etc., so as to make the interpretation of the meaning of discourse more comprehensive and accurate‚ and discover how human beings use multiple modalities to achieve the purpose of social communication. [4]…”

Section: Multimodal Discourse Analysismentioning

confidence: 99%

A Humorous Multimodal Study of the Movie "This is not What I Expected"

2023

artpl

View full text Add to dashboard Cite

Film, as a typical example of multimodal discourse analysis, is a blend of auditory and visual sense, through various means such as language, image, sound, movement and symbols of resources to bring the audience an audio-visual feast. Lu Jin, played by Takeshi Kanagi, represents the superior upper class, while Gu Shengnan, played by Zhou Dongyu, is the representative of the civilian class. Two people with different characters and classes fall in love with each other because of food, which produces scenes of joy that make people laugh. Based on the theory of multimodal discourse analysis and the theory of humor, this paper discusses the mechanism of humor in the movie "This is not what I expected" and analyzes the loving process between the hero and heroine from meeting, knowing, getting along with each other and falling in love. This paper also discusses the synergistic effect of different modalities in the film.

show abstract

Section: Multimodal Discourse Analysismentioning

confidence: 99%

A Humorous Multimodal Study of the Movie "This is not What I Expected"

2023

artpl

View full text Add to dashboard Cite

show abstract

“…Finally, synergistic systems enable a parallel use of modalities and fuse the information from the different sources. For example, the performance of speech recognition systems can be improved by incorporating the speaker's facial expression (Gibbon, Mertins, & Moore, 2000;Sezgin et al, 2009). From a user's perspective, the user might not interact multimodally, since speech is the only active input modality used.…”

Section: Synergistic Systemsmentioning

confidence: 99%

Natural Multimodal Interaction in the Car - Generating Design Support for Speech, Gesture, and Gaze Interaction while Driving

Roider¹

View full text Add to dashboard Cite

Driving a modern car is more than just maneuvering the vehicle on the road. At the same time, drivers want to listen to music, operate the navigation system, compose, and read messages and more. Future cars are turning from simple means for transportation into smart devices on wheels. This trend will continue in the next years together with the advent of automated vehicles. However, technical challenges, legal regulations, and high costs slow down the penetration of automated vehicles. For this reason, a great majority of people will still be driving manually at least for the next decade. Consequently, it must be ensured that all the features of novel infotainment systems can be used easily, efficiently without distracting the driver from the task of driving and still provide a high user experience. A promising approach to cope with this challenge is multimodal in-car interaction. Multimodal interaction basically describes the combination of different input and output modalities for driver-vehicle interaction. Research has pointed out the potential to create a more flexible, efficient, and robust interaction. In addition to that, the integration of natural interaction modalities such as speech, gestures and gaze, the communication with the car could increase the naturalness of the interaction. Based on these advantages, the researcher community in the field of automotive user interfaces has produced several interesting concepts for multimodal interaction in vehicles. The problem is that the resulting insights and recommendations are often easily applicable in the design process of other concepts because they too concrete or very abstract. At the same time, concepts focus on different aspects. Some aim to reduce distraction while others want to increase efficiency or provide a better user experience. This makes it difficult to give overarching recommendations on how to combine natural input modalities while driving. As a consequence, interaction designers of in-vehicle systems are lacking adequate design support that enables them to transfer existing knowledge about the design of multimodal in-vehicle applications to their own concepts. This thesis addresses this gap by providing empirically validated design support for multimodal in-vehicle applications. It starts with a review of existing design support for automotive and multimodal applications. Based on that we report a series of user experiments that investigate various aspects of multimodal in-vehicle interaction with more than 200 participants in lab setups and driving simulators. During these experiments, we assessed the potentials of multimodality while driving, explored how user interfaces can support speech and gestures, and evaluated novel interaction techniques. The insights from these experiments extend existing knowledge from literature in order to create the first pattern collection for multimodal natural in-vehicle interaction. The collection contains 15 patterns that describe solutions for reoccurring problems when combining natural input with speech, gestures, or gaze in the car in a structured way. Finally, we present a prototype of an in-vehicle information system, which demonstrates the application of the proposed patterns and evaluate it in a driving-simulator experiment. This work contributes to field of automotive user interfaces in three ways. First, it presents the first pattern collection for multimodal natural in-vehicle interaction. Second, it illustrates and evaluates interaction techniques that combine speech and gestures with gaze input. Third, it provides empirical results of a series of user experiments that show the effects of multimodal natural interaction on different factors such as driving performance, glance behavior, interaction efficiency, and user experience.

show abstract

Toward Adaptive Information Fusion in Multimodal Systems

Oviatt

2006

Machine Learning for Multimodal Interaction

View full text Add to dashboard Cite

Multimodal contrastive learning train neural networks by levergaing data from heterogeneous sources such as images and text. Yet, many current multimodal learning architectures cannot generalize to an arbitrary number of modalities and need to be hand-constructed. We propose AutoBIND, a novel contrastive learning framework that can learn representations from an arbitrary number of modalites through graph optimization. We evaluate AutoBIND on Alzhiemer's disease detection because it has real-world medical applicability and it contains a broad range of data modalities. We show that AutoBIND outperforms previous methods on this task, highlighting the generalizablility of the approach.Preprint. Under review.

show abstract

Audio-visual and multimodal speech-based systems

Cited by 12 publications

References 0 publications

A Humorous Multimodal Study of the Movie "This is not What I Expected"

A Humorous Multimodal Study of the Movie "This is not What I Expected"

Natural Multimodal Interaction in the Car - Generating Design Support for Speech, Gesture, and Gaze Interaction while Driving

Toward Adaptive Information Fusion in Multimodal Systems

Contact Info

Product

Resources

About