Zero-Shot Event Detection Using Multi-modal Fusion of Weakly Supervised Concepts

Abstract

Current state-of-the-art systems for visual content analysis require large training sets for each class of interest, and performance degrades rapidly with fewer examples. In this paper, we present a general framework for the zeroshot learning problem of performing high-level event detection with no training exemplars, using only textual descriptions. This task goes beyond the traditional zero-shot framework of adapting a given set of classes with training data to unseen classes. We leverage video and image collections with free-form text descriptions from widely available web sources to learn a large bank of concepts, in addition to using several off-the-shelf concept detectors, speech, and video text for representing videos. We utilize natural language processing technologies to generate event description features. The extracted features are then projected to a common high-dimensional space using text expansion, and similarity is computed in this space. We present extensive experimental results on the large TRECVID MED [26] corpus to demonstrate our approach. Our results show that the proposed concept detection methods significantly outperform current attribute classifiers such as Classemes [34], ObjectBank [21], and SUN attributes [28]. Further, we find that fusion, both within as well as between modalities, is crucial for optimal performance.

Learning From Web Videos for Event Classification

“…As shown in Table VI, our results have a mean average precision of 28.79%, which is a significant gain of 6.5% over the state-of-the-art approach [15], which also curates a training set from the internet. We also compare with other recent methods: 20.80% [16], 16.7 [9], 16.1 [7], 11.89% [37], 8.86 [46], 6.39% [10], 6.12% [44], 2.5% [14], 2.3% [3]. We outperform all these methods significantly, due to the following key differences.…”

“…Improvements to this scheme include modeling a pair of words or n-grams to not only exploit the cooccurrences between words [27], but also disambiguate among the multiple meanings represented by individual words [5]. Works in this paradigm, where events are represented as a collection of concept responses, are increasingly leveraging weakly annotated data from the internet [3], [44]. The method in [46] builds a list of frequent words in the text metadata, which represent concepts, to prune videos downloaded from YouTube.…”

“…We compare with several approaches [3], [7], [10], [14]- [16], [37], [44] on the TrecVid 2013 EK0 challenge dataset. Here, we focused on methods which build a training set from internet data.…”

“…We outperform all these methods significantly, due to the following key differences. We use motion information effectively, in contrast to [3], [7], [14], [37], [44], [46], relying on image data alone. Our query expansion method exploits critical elements of actions, e.g., related verbs, whereas [3], [7], [9], [10], [14]- [16], [37], [44], [46] are limited to queries which are not as rich.…”

Semantic Reasoning in Zero Example Video Event Retrieval

“…The advantage of word2vec over other semantic embedding methods is that the latent variables are transparent, because the words are represented in vector space with only a few hundred dimensions. Examples of other semantic embedding methods are Wu et al [2014] with their common lexicon layer and Habibian et al [2014b] with VideoStory and Jain et al [2015] with the embedding of text, actions and objects to classify actions.…”