Active Speaker Detection as a Multi-Objective Optimization with Uncertainty-Based Multimodal Fusion

Pouthier, Baptiste; Pilati, Laurent; Gudupudi, Leela K.; Bouveyron, Charles; Precioso, Fŕed́eric

doi:10.21437/interspeech.2021-80

Cited by 6 publications

(2 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent research has focused on developing new techniques and models to improve ASD performance. Pouthier et al [52] introduced a novel multi-modal fusion scheme based on self-attention and uncertainty to leverage audio and video modalities for ASD. Similarly, Kopuklu et al [53] proposed a pipeline consisting of audio-visual encoding, inter-speaker modeling and temporal modeling stages, known as ASDNet, for detecting active speakers in challenging environments.…”

Section: Active Speaker Detection (Asd)mentioning

confidence: 99%

Enhancing Human–Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition

Paul

Nicolescu

2023

Sensors

View full text Add to dashboard Cite

With the increasing presence of robots in our daily lives, it is crucial to design interaction interfaces that are natural, easy to use and meaningful for robotic tasks. This is important not only to enhance the user experience but also to increase the task reliability by providing supplementary information. Motivated by this, we propose a multi-modal framework consisting of multiple independent modules. These modules take advantage of multiple sensors (e.g., image, sound, depth) and can be used separately or in combination for effective human–robot collaborative interaction. We identified and implemented four key components of an effective human robot collaborative setting, which included determining object location and pose, extracting intricate information from verbal instructions, resolving user(s) of interest (UOI), and gesture recognition and gaze estimation to facilitate the natural and intuitive interactions. The system uses a feature–detector–descriptor approach for object recognition and a homography-based technique for planar pose estimation and a deep multi-task learning model to extract intricate task parameters from verbal communication. The user of interest (UOI) is detected by estimating the facing state and active speakers. The framework also includes gesture detection and gaze estimation modules, which are combined with a verbal instruction component to form structured commands for robotic entities. Experiments were conducted to assess the performance of these interaction interfaces, and the results demonstrated the effectiveness of the approach.

show abstract

Section: Active Speaker Detection (Asd)mentioning

confidence: 99%

Enhancing Human–Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition

Paul

Nicolescu

2023

Sensors

View full text Add to dashboard Cite

show abstract

“…However, earlier efforts on ASD were limited to short sequences of frontal faces, which did not accurately reflect the complexities of real-world situations [8,9,10]. With the release of the AVA-ActiveSpeaker dataset [13], which is the first large-scale standard benchmark for ASD tasks, several high-performing large networks have been proposed to model various types of relational information in audiovisual modalities [11,12,14,15,32]. For example, the Unified Context Network (UniCon) [16] achieved superior performance by introducing a relational context module to intensively capture the spatial relationships between all speakers in a video frame.…”

Section: Introductionmentioning

confidence: 99%

Modeling Long-Term Multimodal Representations for Active Speaker Detection With Spatio-Positional Encoder

Kyoung,

Song

2023

IEEE Access

View full text Add to dashboard Cite

In this study, we present an end-to-end framework for active speaker detection to achieve robust performance in challenging scenarios with multiple speakers. In contrast to recent approaches, which rely heavily on the visual relational context between all speakers in a video frame, we propose collaboratively learning multimodal representations based on the audio and visual signals of a single candidate. Firstly, we propose a spatio-positional encoder to effectively address the problem of false detections caused by indiscernible faces in a video frame. Secondly, we present an efficient multimodal approach that models the long-term temporal contextual interactions between audio and visual modalities. Through extensive experiments on the AVA-ActiveSpeaker dataset, we demonstrate that our framework notably outperforms recent state-of-the-art approaches under challenging multi-speaker settings. Additionally, the proposed framework significantly improves the robustness against auditory and visual noise interference without relying on pre-trained networks or hand-crafted training strategies.INDEX TERMS Active speaker detection, audio-visual, multimodal representations, multi-speaker.

show abstract

Asd-Transformer: Efficient Active Speaker Detection Using Self And Multimodal Transformers

Datta

Etchart

Yadav

et al. 2022

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Active Speaker Detection as a Multi-Objective Optimization with Uncertainty-Based Multimodal Fusion

Cited by 6 publications

References 17 publications

Enhancing Human–Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition

Enhancing Human–Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition

Modeling Long-Term Multimodal Representations for Active Speaker Detection With Spatio-Positional Encoder

Asd-Transformer: Efficient Active Speaker Detection Using Self And Multimodal Transformers

Contact Info

Product

Resources

About