Video2vec Embeddings Recognize Events When Examples Are Scarce

Abstract

Abstract-This paper aims for event recognition when video examples are scarce or even completely absent. The key in such a challenging setting is a semantic video representation. Rather than building the representation from individual attribute detectors and their annotations, we propose to learn the entire representation from freely available web videos and their descriptions using an embedding between video features and term vectors. In our proposed embedding, which we call Video2vec, the correlations between the words are utilized to learn a more effective representation by optimizing a joint objective balancing descriptiveness and predictability. We show how learning the Video2vec using a multimodal predictability loss, including appearance, motion and audio features, results in a better predictable representation. We also propose an event specific variant of Video2vec to learn a more accurate representation for the words, which are indicative of the event, by introducing a term sensitive descriptiveness loss. Our experiments on three challenging collections of web videos from the NIST TRECVID Multimedia Event Detection and Columbia Consumer Videos datasets demonstrate: i) the advantages of Video2vec over representations using attributes or alternative embeddings, ii) the benefit of fusing video modalities by an embedding over common strategies, iii) the complementarity of term sensitive descriptiveness and multimodal predictability for event recognition. By its ability to improve predictability of present day audio-visual video features, while at the same time maximizing their semantic descriptiveness, Video2vec leads to state-of-the-art accuracy for both few-and zero-example recognition of events in video.

Semantic Pooling for Complex Event Analysis in Untrimmed Videos

“…It is convenient because it is fully automatic. We note that the recent work [58] proposed a different approach for event detection when the number of labeled training examples is limited. A potentially superior approach is to extract relevant keyframes from the positive training exemplars and use these to define saliency.…”

Unified Embedding and Metric Learning for Zero-Exemplar Event Detection

“…In test time, video ranking and retrieval is done using distance metric between the projected test video y t and test query representation y. [16,17] project the visual feature x of a web video v into term-vector representation y of the video's textual title t. However, during training, the model makes use of the text query of the test events to learn better term-vector representation. Consequently, this limits the generalization for novel event queries.…”

“…In the middle, we borrow the network fT to embed the event article's feature y t as z t ∈ Z. Then, at the bottom, the network fV learns to embed the video feature x as z v ∈ Z such that the distance between z v , z t is minimized, in the learned metric space Z. self-paced reranking [23], pseudo-relevance feedback [24], event query manual intervention [25], early fusion of features (action [26,27,28,29,30] or acoustic [31,32,33]) or late fusion of concept scores [17]. All these contributions may be applied to our method.…”

“…Event detection accuracies: per-event average precision (AP%) and per-dataset mean average precision (mAP%) for MED-13 and MED-14 datasets. We compare our results against TagBook [18], Discovary [7], Composition [8], Classifiers [9] and VideoStory [17].…”

“…However, VideoStory * uses motion feature representation and expert text query (i.e. using term-importance matrix H in [17]). To rule out the marginal effect of using different datasets and features, we train VideoStory and report results in table 3.…”