Multimodal Speaker Diarization Utilizing Face Clustering Information

Abstract

Abstract. Multimodal clustering/diarization tries to answer the question "who spoke when" by using audio and visual information. Diarization consists of two steps, at first segmentation of the audio information and detection of the speech segments and then clustering of the speech segments to group the speakers. This task has been mainly studied on audiovisual data from meetings, news broadcasts or talk shows. In this paper, we use visual information to aid speaker clustering. We tested the proposed method in three full length movies, i.e. a scenario much more difficult than the ones used so far, where there is no certainty that speech segments and video appearances of actors will always overlap. The results proved that the visual information can improve the speaker clustering accuracy and hence the diarization process.

Speaker Naming in Movies

“…Other studies focused on the problem of speaker recognition without naming, using the speech modality as a single source of information. While some of these studies attempted to incorporate the visual modality, their goal was to cluster the speech segments rather than name the speakers arXiv:1809.08761v1 [cs.CL] 24 Sep 2018 (Erzin et al, 2005;Bost and Linares, 2014;Kapsouras et al, 2015;Bredin and Gelly, 2016;Hu et al, 2015;Ren et al, 2016). None of these studies used textual information (e.g., dialogue), which prevented them from identifying speaker names.…”