Name-It: naming and detecting faces in news videos

Abstract

We developed NameIt, a system that associates faces and names in news videos. It processes information from the videos and can infer possible name candidates for a given face or locate a face in news videos by name. To accomplish this task, the system takes a multimodal video analysis approach: face sequence extraction and similarity evaluation from videos, name extraction from transcripts, and video-caption recognition.T he Name-It system 1,2 associates names and faces in news videos. Assume that we're watching a TV news program. When persons we don't know appear in the news video, we can eventually identify most of them by watching only the video. To do this, we detect faces from a news video, locate names in the sound track, and then associate each face to the correct name. For face-name association, we use as many hints as possible based on structure, context, and meaning of the news video. We don't need any additional knowledge such as newspapers containing descriptions of the persons or biographical dictionaries with pictures. Similarly, Name-It can associate faces in news videos with their right names without using an a priori face-name association set. In other words, Name-It extracts face-name correspondences only from news videos.Name-It takes a multimodal approach to accomplish this task. For example, it uses several information sources available from news videosimage sequences, transcripts, and video captions. Name-It detects face sequences from image sequences and extracts name candidates from transcripts. It's possible to obtain transcripts from audio tracks by using the proper speech recognition technique with an allowance for recognition errors. However, most news broadcasts in the US already have closed captions. (In the near future, the worldwide trend will be for broadcasts to feature closed captions.) Thus we use closed-caption texts as transcripts for news videos. In addition, we employ video-caption detection and recognition. We used "CNN Headline News" as our primary source of news for our experiments.Given image sequences, transcripts, and video captions as information sources, Name-It associates extracted faces with extracted name candidates using the correlation of their timing information and face similarity information. Video captions are also taken into account as supplementary information. To associate faces and names, Name-It integrates several advanced image processing and natural-language processing techniques-face sequence extraction and similarity evaluation from videos, name extraction from transcripts, and video-caption recognition. Although these technologies aren't always highly accurate, integrating these results will help the system achieve more accurate output.With respect to face-name association, the Piction system 3 works similarly to Name-It. Piction identifies faces within a given captioned newspaper photograph by extracting faces from the photograph and analyzing the caption to obtain geometric constraints among faces. The system then labels each face with a name. ...

Face Recognition and Retrieval in Video

“…The dramatic increase of videos demands more efficient and accurate access to video content. Finding a specific person in videos is essential to understand and retrieve videos [91]. Face information is an important cue for person identification in many types of videos such as news programs, dramas, movies, and homemade videos.…”

Event Mining in Multimedia Streams

“…Explicit labeling for who [12], [16], [119] or where [156] exploit the correspondences between the visual information and the spoken content.…”

Automatic Detection and Clustering of Actor Faces based on Spectral Clustering Techniques

“…In [17], faces in news videos are recognized and retrieved on the basis of visual and textual information from transcripts. In [20], a spectral clustering algorithm, called the Ng algorithm [1], is used to cluster a relatively small set of faces (39 faces for 6 peoples) described by their eigenface projection.…”

“…While automatic face recognition aims at matching observed faces against a face database [8] [9][12] [17] [21], automatic casting consists in forming groups of similar faces represented by keyfaces (i.e. principal actors) [12].…”