Face Recognition from Video Using the Generic Shape-Illumination Manifold

Abstract

Abstract. In spite of over two decades of intense research, illumination and pose invariance remain prohibitively challenging aspects of face recognition for most practical applications. The objective of this work is to recognize faces using video sequences both for training and recognition input, in a realistic, unconstrained setup in which lighting, pose and user motion pattern have a wide variability and face images are of low resolution. In particular there are three areas of novelty: (i) we show how a photometric model of image formation can be combined with a statistical model of generic face appearance variation, learnt offline, to generalize in the presence of extreme illumination changes; (ii) we use the smoothness of geodesically local appearance manifold structure and a robust same-identity likelihood to achieve invariance to unseen head poses; and (iii) we introduce an accurate video sequence "reillumination" algorithm to achieve robustness to face motion patterns in video. We describe a fully automatic recognition system based on the proposed method and an extensive evaluation on 171 individuals and over 1300 video sequences with extreme illumination, pose and head motion variation. On this challenging data set our system consistently demonstrated a nearly perfect recognition rate (over 99.7% on all three databases), significantly outperforming state-of-the-art commercial software and methods from the literature.

Illumination Processing in Face Recognition

“…Arandjelović and Cipolla (Arandjelović & Cipolla, 2006) proposed the Generic Shape-Illumination (gSIM) algorithm. The authors showed how a photometric model of image formation can be combined with a statistical model of generic face appearance variation, learnt offline, to generalize in the presence of extreme illumination changes.…”

Dictionary-Based Face Recognition from Video

“…In a related work, Arandjelovic and Cipolla [13] represent the appearance variations due to shape and illumination on faces by assuming that the shapeillumination manifold of all possible illuminations and poses is generic for faces. This in turn implies that the shape-illumination manifold can be estimated using a set of subjects independent of the test set.…”

Multi-Eigenspace Learning for Video-Based Face Recognition

“…They also exploit audio information in videos. In [1], a generic shape-illumination manifold is learnt off-line using Probabilistic PCA (PPCA) [15]. A generic framework for both tracking and recognition is proposed by Zhou et al [19,20].…”

“…Initially, k is assigned as 1 and the center of Ω (1) is assigned as the first frame. Then for each incoming frame x t , compute the distance between x t and each eigenspace center.…”

“…For video-based face recognition, most state-of-the-art algorithms [1,2,5,9,10,12] can perform the recognition task in real time . However, the training process commonly runs off-line in a batch mode.…”