Automatic propagation of manual annotations for multimodal person identification in TV shows

Abstract

Abstract-In this paper an approach to human annotation propagation for person identification in the multimodal context is proposed. A system is used, which combines speaker diarization and face clustering to produce multimodal clusters. The whole multimodal clusters are later annotated rather than just single tracks, which is done by propagation. Optical character recognition systems provides initial annotation. Four different strategies, which select candidates for annotation, are tested. The initial results of annotation propagation are promising. With the use of a proper active learning selection strategy the human annotator involvement could be reduced even further.

Learnable PINs: Cross-Modal Embeddings for Person Identity

“…This is di cult because of the significant visual variation of character appearances in a TV show caused by pose, illumination, size, expression and occlusion, which can often exceed those due to identity. Recently there has been a growing interest in the use of the audio-track to aid identification [9,36,47] which comes for free with multimedia videos. However, because face and voice representations are usually not aligned, in prior work the query face cannot be directly compared to the audio track, necessitating the use of complex fusion systems to combine information from both modalites.…”

“…However, because face and voice representations are usually not aligned, in prior work the query face cannot be directly compared to the audio track, necessitating the use of complex fusion systems to combine information from both modalites. For example, [9] use clustering on face-tracks and diarised speaker segments after a round of human annotation for both, [36] use confidence labels from one modality to provide supervsion for the other modality, and [47] fuse the outputs of a face recognition model, and a clothing model, with a GMM-based speaker model. With a joint embedding, however, the query face image can be compared directly to the audio track, leading to an extremely simple solution which we describe below.…”

Segmentation et Regroupement en Locuteurs: comment évaluer les corrections humaines

“…Dans Bazillon et al (2008), les auteurs ont montré que la correction des sorties d'un système de transcription automatique de la parole permet de diminuer le temps d'annotation. Une méthode d'apprentissage actif (active learning en anglais), proposée dans Budnik et al (2014), utilisée en conjonction avec des systèmes de reconnaissance automatique du locuteur et du visage, réduit davantage le nombre d'interactions homme-machine. Récemment, dans Broux et al (2016), nous avons proposé un système qui assiste la SRL et réduit le nombre d'interventions humaines.…”