The authors describe a face tracking and recognition system for video indexing that handles variable face poses (left-right and up-down) and deformations due to speech and facial expressions. The system is based on deformable template matching, and employs person-specific templates at near-frontal poses for recognition, and novel person-independent templates at multiple poses on the view-sphere for tracking. Relative to an earlier version that used multiple person-specific templates at multiple (left-right) poses, the new system speeds up processing by (i) restricting attention to skin-color regions; (ii) performing recognition using the person-specific templates at near-frontal poses only; and (iii) tracking at non-frontal poses using the novel person-independent templates. Registration is also simplified since multiple views of each target individual are no longer required, but at the cost of a loss of recognition functionality at poses far from frontal (the system instead “remembers” the identity of each individual from near-frontal matches and tracks between them).