Clip retrieval using multi-modal biometrics in meeting archives

Abstract

We present a system to retrieve all clips from a meeting archive that show a particular individual speaking, using a single face or voice sample as the query. The system incorporates three novel ideas. One, rather than match the query to each individual sample in the archive, samples within a meeting are grouped first, generating a cluster of samples per individual. The query is then matched to the cluster, taking advantage of multiple samples to yield a robust decision. Two, automatic audio-visual association is performed which allows a bi-modal retrieval of clips, even when the query is uni-modal. Three, the biometric recognition uses individual-specific score distributions learnt from the clusters, in a likelihood ratio based decision framework that obviates the need for explicit normalization or modality weighting. The resulting system, which is completely automated, performs with 92.6% precision at 90% recall on a dataset of 16 real meetings spanning a total of 13 hours.

Increasing Importance of Joint Analysis of Audio and Video in Computer Vision: A Survey

“…This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2024.3391817 recognition [31], movie genre recognition [31], [32], video summarization [40], tracking [5], [21], [22], [33], video search & retrieval [36].…”

“…Vajaria et al proposed various approaches for speaker localization [35] and used them to solve speaker clip retrieval [36] and speaker diarization [37]. They used the Bayesian information criterion to segment feature vectors and graph spectral partitioning to cluster the segments of a speaker in a video clip.…”