Naming TV characters by watching and analyzing dialogs

Haurilet, Monica-Laura; Tapaswi, Makarand; Al-Halah, Ziad; Stiefelhagen, Rainer

doi:10.1109/wacv.2016.7477560

Cited by 31 publications

(29 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Face track clustering/identification methods have also used additional cues such as clothing appearance [33], speech [22], voice models [19], context [46], gender [49], name mentions (first, second, and third person references) in subtitles [18], weak labels using transcripts/subtitles [7], [2], and joint action and actor labeling [16] using transcripts.…”

Section: Related Workmentioning

confidence: 99%

Self-Supervised Learning of Face Representations for Video Face Clustering

Sharma

Tapaswi

Sarfraz

et al. 2019

2019 14th IEEE International Conference on Automatic Face &Amp; Gesture Recognition (FG 2019)

Self Cite

View full text Add to dashboard Cite

Analyzing the story behind TV series and movies often requires understanding who the characters are and what they are doing. With improving deep face models, this may seem like a solved problem. However, as face detectors get better, clustering/identification needs to be revisited to address increasing diversity in facial appearance. In this paper, we address video face clustering using unsupervised methods. Our emphasis is on distilling the essential information, identity, from the representations obtained using deep pre-trained face networks. We propose a self-supervised Siamese network that can be trained without the need for video/track based supervision, and thus can also be applied to image collections. We evaluate our proposed method on three video face clustering datasets. The experiments show that our methods outperform current state-of-the-art methods on all datasets. Video face clustering is lacking a common benchmark as current works are often evaluated with different metrics and/or different sets of face tracks. The datasets and code are available at https://github.com/vivoutlaw/SSIAM.

show abstract

Section: Related Workmentioning

confidence: 99%

Self-Supervised Learning of Face Representations for Video Face Clustering

Sharma

Tapaswi

Sarfraz

et al. 2019

2019 14th IEEE International Conference on Automatic Face &Amp; Gesture Recognition (FG 2019)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Verification losses. Next, we analyze LDML, contrastive, and triplet losses ( Table 5 rows [10][11][12][13][14][15][16][17][18]. While these losses are often used to perform clustering, they are not designed for it [47].…”

Section: #Chmentioning

confidence: 99%

“…Characters are often studied by analyzing face tracks (sequences of temporally related detections) in videos. A significant part of this analysis is identification -labeling face tracks with their names, and typically employs supervision from web images [1,29], transcripts [3,9], or even dialogs [7,15]. We are interested in an equally popular alternative -clustering face tracks based on identity.…”

Section: Introductionmentioning

confidence: 99%

Video Face Clustering With Unknown Number of Clusters

Tapaswi

Law²,

Fidler

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

View full text Add to dashboard Cite

Understanding videos such as TV series and movies requires analyzing who the characters are and what they are doing. We address the challenging problem of clustering face tracks based on their identity. Different from previous work in this area, we choose to operate in a realistic and difficult setting where: (i) the number of characters is not known a priori; and (ii) face tracks belonging to minor or background characters are not discarded.To this end, we propose Ball Cluster Learning (BCL), a supervised approach to carve the embedding space into balls of equal size, one for each cluster. The learned ball radius is easily translated to a stopping criterion for iterative merging algorithms. This gives BCL the ability to estimate the number of clusters as well as their assignment, achieving promising results on commonly used datasets. We also present a thorough discussion of how existing metric learning literature can be adapted for this task. 1 We consider three types of characters based on their roles: primary or recurring characters have major roles in several episodes; secondary or minor characters are named and play an important role in some episodes; and background or unknown (Unk) characters are unnamed and uncredited.

show abstract

“…Weak Supervision for Actor Labelling The topic of labeling actors in movies and TV-series has been tackled mostly using weak supervision from subtitle and transcript files [1,8,9,10,11,12,13], building on the seminal work of Everingham et al [1,8]. The transcript tells us what each character says in the show, but is not aligned with the video.…”

Section: Related Workmentioning

confidence: 99%

Who’s that Actor? Automatic Labelling of Actors in TV Series Starting from IMDB Images

Aljundi

Chakravarty

Tuytelaars

2017

Computer Vision – ACCV 2016

View full text Add to dashboard Cite

Abstract. In this work, we aim at automatically labeling actors in a TV series. Rather than relying on transcripts and subtitles, as has been demonstrated in the past, we show how to achieve this goal starting from a set of example images of each of the main actors involved, collected from the Internet Movie Database (IMDB). The problem then becomes one of domain adaptation: actors' IMDB photos are typically taken at awards ceremonies and are quite different from their appearances in TV series. In each series as well, there is considerable change in actor appearance due to makeup, lighting, ageing, etc. To bridge this gap, we propose a graph-matching based self-labelling algorithm, which we coin HSL (Hungarian Self Labeling). Further, we propose a new edge cost to be used in this context, as well as an extension that is more robust to outliers, where prototypical faces for each of the actors are selected based on a hierarchical clustering procedure. We conduct experiments with 15 episodes from 3 different TV series and demonstrate automatic annotation with an accuracy of 90% and up.

show abstract

Naming TV characters by watching and analyzing dialogs

Cited by 31 publications

References 28 publications

Self-Supervised Learning of Face Representations for Video Face Clustering

Self-Supervised Learning of Face Representations for Video Face Clustering

Video Face Clustering With Unknown Number of Clusters

Who’s that Actor? Automatic Labelling of Actors in TV Series Starting from IMDB Images

Contact Info

Product

Resources

About