A Generative Approach to Zero-Shot and Few-Shot Action Recognition

Abstract

We present a generative framework for zero-shot action recognition where some of the possible action classes do not occur in the training data. Our approach is based on modeling each action class using a probability distribution whose parameters are functions of the attribute vector representing that action class. In particular, we assume that the distribution parameters for any action class in the visual space can be expressed as a linear combination of a set of basis vectors where the combination weights are given by the attributes of the action class. These basis vectors can be learned solely using labeled data from the known (i.e., previously seen) action classes, and can then be used to predict the parameters of the probability distributions of unseen action classes. We consider two settings: (1) Inductive setting, where we use only the labeled examples of the seen action classes to predict the unseen action class parameters; and (2) Transductive setting which further leverages unlabeled data from the unseen action classes. Our framework also naturally extends to few-shot action recognition where a few labelled examples from unseen classes are available. Our experiments on benchmark datasets (UCF101, HMDB51 and Olympic) show significant performance improvements as compared to various baselines, in both standard zero-shot (disjoint seen and unseen classes) and generalized zero-shot learning settings.

Out-Of-Distribution Detection for Generalized Zero-Shot Action Recognition

“…The best existing approach for GZSL action recognition, GGM [24], employs a generative approach to synthesize unseen class data and utilizes unlabelled real features (C3D) from the unseen classes to rectify the bias of the learned parameters towards seen classes. Particularly, for the UCF101 dataset and manual attributes combination, the proposed approach, CEWGAN-OD, achieves gains of 5.1% and 25.8% (in terms of accuracy) over the CLSWGAN [33] and GGM [24], respectively. Further, for the word2vec embedding, the proposed CEWGAN-OD achieves gains of 16% and 19.8% over the best existing approach, GGM [24], for the HMDB51 and UCF101 datasets, respectively.…”

“…Particularly, for the UCF101 dataset and manual attributes combination, the proposed approach, CEWGAN-OD, achieves gains of 5.1% and 25.8% (in terms of accuracy) over the CLSWGAN [33] and GGM [24], respectively. Further, for the word2vec embedding, the proposed CEWGAN-OD achieves gains of 16% and 19.8% over the best existing approach, GGM [24], for the HMDB51 and UCF101 datasets, respectively. ZSL performance comparison: In Tab.…”

Generalized Zero-Shot Learning using Generated Proxy Unseen Samples and Entropy Separation

“…In the standard GZSL setting, we improve by 9.3 and 4.9 in over the non-generative model SADLE in [28], for HMDB51 and UCF101, respectively. Generativemodel driven methods GGM [20], f-CLSWGAN [35] and CEWGAN [16] uses unseen class prototypes during training to generate unseen class visual samples. We still outperform GGM, f-CLSWGAN, and deliver comparable performance with CEWGAN.…”

Rethinking Zero-Shot Video Classification: End-to-End Training for Realistic Applications

“…We focus on inductive ZSL in which test data is fully unknown at training time. There exists a body of literature on transductive ZSL [1,33,54,55,59,58,60], where test images or videos are available during training but test labels are not. We do not discuss the transductive approach in this work.…”

“…To our knowledge, all current ZSL methods for video recognition use pretrained visual embeddings [1,4,18,33,35,54,55,58,59,60,61,64]. This provides a good tradeoff between training efficiency and using prior knowledge.…”