Self-Supervised Learning of Face Representations for Video Face Clustering

Sharma, Vivek; Tapaswi, Makarand; Sarfraz, M. Saquib; Stiefelhagen, Rainer

doi:10.1109/fg.2019.8756609

Cited by 33 publications

(52 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, it is worth noting the self-supervised learning works on "harvesting" training data from unlabeled sources for action recognition. Fernando et al [12] and Mishra et al [28] shuffle the video frames and treat them as positive/negative training data; Sharma et al [34] mines labels using a distance matrix based on similarity although for video face clustering; Wei et al [51] divides a single clip into non-overlapping 10-frame chunks, and then predict the ordering task; Ng et al [29] estimates optical flow while recognizing actions. We compare all these methods against our unsupervised future frame prediction based ConvNet training in the experimental section.…”

Section: Background and Related Workmentioning

confidence: 99%

DynamoNet: Dynamic Action and Motion Network

Diba

Sharma²,

Gool

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

115

View full text Add to dashboard Cite

In this paper, we are interested in self-supervised learning the motion cues in videos using dynamic motion filters for a better motion representation to finally boost human action recognition in particular. Thus far, the vision community has focused on spatio-temporal approaches using standard filters, rather we here propose dynamic filters that adaptively learn the video-specific internal motion representation by predicting the short-term future frames. We name this new motion representation, as dynamic motion representation (DMR) and is embedded inside of 3D convolutional network as a new layer, which captures the visual appearance and motion dynamics throughout entire video clip via end-to-end network learning. Simultaneously, we utilize these motion representation to enrich video classification. We have designed the frame prediction task as an auxiliary task to empower the classification problem.With these overall objectives, to this end, we introduce a novel unified spatio-temporal 3D-CNN architecture (Dy-namoNet) that jointly optimizes the video classification and learning motion representation by predicting future frames as a multi-task learning problem. We conduct experiments on challenging human action datasets: Kinetics 400, UCF101, HMDB51. The experiments using the proposed DynamoNet show promising results on all the datasets.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

DynamoNet: Dynamic Action and Motion Network

Diba

Sharma²,

Gool

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

115

View full text Add to dashboard Cite

show abstract

“…Most of these studies attempt to utilize redundant information of face sequences/sets to improve recognition performance, but not to learn discriminative features from sequence data. Recently, some approaches [34,[36][37][38][39] aim to learn deep video features for video face recognition. In [37], large-scale unlabeled face sequences are employed as the training data, but these sequence data are only utilized to learn transformations between image and video domains.…”

Section: Sequences In Face Recognitionmentioning

confidence: 99%

“…In [37], large‐scale unlabeled face sequences are employed as the training data, but these sequence data are only utilized to learn transformations between image and video domains. [38] trained a self‐supervised Siamese network to obtain the features of a face cluster instead of a single face. [39] also aims to cluster faces in videos more accurately.…”

Section: Related Workmentioning

confidence: 99%

SeqFace: Learning discriminative features by using face sequences

Huang

Zhang

et al. 2021

IET Image Processing

View full text Add to dashboard Cite

Deep convolutional neural networks (CNNs) have greatly improved the Face Recognition (FR) performance in recent years. Almost all CNNs in FR are trained on the carefully labeled datasets containing plenty of identities. However, such high‐quality datasets are very expensive to collect, which restricts many researchers to achieve state‐of‐the‐art performance. In this paper, a framework, called SeqFace, for learning discriminative face features is proposed. Besides a traditional identity training dataset, the designed SeqFace can train CNNs by using an additional dataset which includes a large number of face sequences collected from videos. Moreover, the label smoothing regularization (LSR) and a new proposed discriminative sequence agent (DSA) loss are employed to enhance the discrimination power of deep face features via making full use of the sequence data. Only with a single ResNet model, the method achieves very competitive performance on several face recognition benchmarks, including LFW, YTF, CFP, AgeDB, and MegaFace. The code and model are publicly available at the website https://github.com/huangyangyu/SeqFace.

show abstract

“…Ignoring tracks, metrics are learned by ranking a batch of frames and creating hard positive and negative pairs [39]. However, all of the above methods require knowledge of the number of clusters K that is difficult to estimate beforehand; and only consider primary characters (tracks for background characters are ignored).…”

Section: Related Workmentioning

confidence: 99%

Video Face Clustering With Unknown Number of Clusters

Tapaswi

Law²,

Fidler

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

View full text Add to dashboard Cite

Understanding videos such as TV series and movies requires analyzing who the characters are and what they are doing. We address the challenging problem of clustering face tracks based on their identity. Different from previous work in this area, we choose to operate in a realistic and difficult setting where: (i) the number of characters is not known a priori; and (ii) face tracks belonging to minor or background characters are not discarded.To this end, we propose Ball Cluster Learning (BCL), a supervised approach to carve the embedding space into balls of equal size, one for each cluster. The learned ball radius is easily translated to a stopping criterion for iterative merging algorithms. This gives BCL the ability to estimate the number of clusters as well as their assignment, achieving promising results on commonly used datasets. We also present a thorough discussion of how existing metric learning literature can be adapted for this task. 1 We consider three types of characters based on their roles: primary or recurring characters have major roles in several episodes; secondary or minor characters are named and play an important role in some episodes; and background or unknown (Unk) characters are unnamed and uncredited.

show abstract

Self-Supervised Learning of Face Representations for Video Face Clustering

Cited by 33 publications

References 48 publications

DynamoNet: Dynamic Action and Motion Network

DynamoNet: Dynamic Action and Motion Network

SeqFace: Learning discriminative features by using face sequences

Video Face Clustering With Unknown Number of Clusters

Contact Info

Product

Resources

About