Who Said That?: Audio-Visual Speaker Diarisation of Real-World Meetings

Chung, Joon Son; Lee, Bong-Jin; Han, Icksang

doi:10.21437/interspeech.2019-3116

Cited by 33 publications

(29 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, several methods that leverage audio and visual cues for diarization are motivated by the synergy between utterances and lip movements. These methods adopt techniques such as mutual information [26,47], canonical correlation analysis [28], and deep learning [10,12,13]. In recent works, audio-visual correspondence is also used for associating talking faces and voice tracks [9,60,67].…”

Section: Related Workmentioning

confidence: 99%

“…Thus, completely off-screen speakers deserve further investigation. While early work in speaker diarization solely depends on the audio stream [54,64,71], recent works [9,10,22] start to attempt leveraging both audio and visual cues. These methods either fuse the similarity scores of two independent modalities or incorporate the synchronization between utterance and lip motion.…”

Section: Introductionmentioning

confidence: 99%

“…This failure makes these methods less robust to the off-screen speakers. The recent work [10] runs an audiovisual synchronization model offline and enroll speakers in advance based on the profiles provided by the users, but this paradigm would fail to enroll the invisible speakers.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

AVA-AVD: Audio-Visual Speaker Diarization in the Wild

Xu,

Song,

Tsutsui

et al. 2021

Preprint

View full text Add to dashboard Cite

Audio-visual speaker diarization aims at detecting "who spoken when" using both auditory and visual signals. Existing audio-visual diarization datasets are mainly focused on indoor environments like meeting rooms or news studios, which are quite different from in-the-wild videos in many scenarios such as movies, documentaries, and audience sitcoms. To create a testbed that can effectively compare diarization methods on videos in the wild, we annotate the speaker diarization labels on the AVA movie dataset and create a new benchmark called AVA-AVD. This benchmark is challenging due to the diverse scenes, complicated acoustic conditions, and completely off-screen speakers. Yet, how to deal with off-screen and on-screen speakers together still remains a critical challenge. To overcome it, we propose a novel Audio-Visual Relation Network (AVR-Net) which introduces an effective modality mask to capture discriminative information based on visibility. Experiments have shown that our method not only can outperform state-of-the-art methods but also is more robust as varying the ratio of off-screen speakers. Ablation studies demonstrate the advantages of the proposed AVR-Net and especially the modality mask on diarization. Our data and code will be made publicly available at https://github.com/zcxueric/AVA-AVD.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

AVA-AVD: Audio-Visual Speaker Diarization in the Wild

Xu,

Song,

Tsutsui

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In addition to text, vision can be combined with speech as well. Such tasks include audio-visual speech recognition [50]- [52], speaker recognition [53]- [55], as well as speech diarisation [56], [57], separation [58], [59] and enhancement [60], which mostly focused on the use of visual features to improve the robustness of the audio-only methods.…”

Section: Introductionmentioning

confidence: 99%

Multimodal Intelligence: Representation Learning, Information Fusion, and Applications

Zhang

Yang

et al. 2020

IEEE J. Sel. Top. Signal Process.

288

View full text Add to dashboard Cite

Deep learning has revolutionized speech recognition, image recognition, and natural language processing since 2010, each involving a single modality in the input signal. However, many applications in artificial intelligence involve more than one modality. It is therefore of broad interest to study the more difficult and complex problem of modeling and learning across multiple modalities. In this paper, a technical review of the models and learning methods for multimodal intelligence is provided. The main focus is the combination of vision and natural language, which has become an important area in both computer vision and natural language processing research communities.This review provides a comprehensive analysis of recent work on multimodal deep learning from three new angles -learning multimodal representations, the fusion of multimodal signals at various levels, and multimodal applications. On multimodal representation learning, we review the key concept of embedding, which unifies the multimodal signals into the same vector space and thus enables cross-modality signal processing. We also review the properties of the many types of embedding constructed and learned for general downstream tasks. On multimodal fusion, this review focuses on special architectures for the integration of the representation of unimodal signals for a particular task. On applications, selected areas of a broad interest in current literature are covered, including caption generation, text-to-image generation, and visual question answering. We believe this review can facilitate future studies in the emerging field of multimodal intelligence for the community.

show abstract

“…Previously, the frame-level portion of these extractors was based on TDNN (time delay neural network) blocks that contained only five convolutional layers with temporal context [6]. Such types of embeddings, referred to as x-vectors, are often applied to diarization tasks in state-of-the-art systems [12]. The newer ECAPA-TDNN (emphasized channel attention, propagation and aggregation TDNN) [13] architecture develops the idea of TDNN and contains additional blocks with hierarchical filters for the extraction of different scale features.…”

Section: Introductionmentioning

confidence: 99%

Application of Fusion of Various Spontaneous Speech Analytics Methods for Improving Far-Field Neural-Based Diarization

et al. 2021

View full text Add to dashboard Cite

Recently developed methods in spontaneous speech analytics require the use of speaker separation based on audio data, referred to as diarization. It is applied to widespread use cases, such as meeting transcription based on recordings from distant microphones and the extraction of the target speaker’s voice profiles from noisy audio. However, speech recognition and analysis can be hindered by background and point-source noise, overlapping speech, and reverberation, which all affect diarization quality in conjunction with each other. To compensate for the impact of these factors, there are a variety of supportive speech analytics methods, such as quality assessments in terms of SNR and RT60 reverberation time metrics, overlapping speech detection, instant speaker number estimation, etc. The improvements in speaker verification methods have benefits in the area of speaker separation as well. This paper introduces several approaches aimed towards improving diarization system quality. The presented experimental results demonstrate the possibility of refining initial speaker labels from neural-based VAD data by means of fusion with labels from quality estimation models, overlapping speech detectors, and speaker number estimation models, which contain CNN and LSTM modules. Such fusing approaches allow us to significantly decrease DER values compared to standalone VAD methods. Cases of ideal VAD labeling are utilized to show the positive impact of ResNet-101 neural networks on diarization quality in comparison with basic x-vectors and ECAPA-TDNN architectures trained on 8 kHz data. Moreover, this paper highlights the advantage of spectral clustering over other clustering methods applied to diarization. The overall quality of diarization is improved at all stages of the pipeline, and the combination of various speech analytics methods makes a significant contribution to the improvement of diarization quality.

show abstract

Who Said That?: Audio-Visual Speaker Diarisation of Real-World Meetings

Cited by 33 publications

References 30 publications

AVA-AVD: Audio-Visual Speaker Diarization in the Wild

AVA-AVD: Audio-Visual Speaker Diarization in the Wild

Multimodal Intelligence: Representation Learning, Information Fusion, and Applications

Application of Fusion of Various Spontaneous Speech Analytics Methods for Improving Far-Field Neural-Based Diarization

Contact Info

Product

Resources

About