HiTIM: Hierarchical Task Information Mining for Few-Shot Action Recognition

Abstract

Although the existing few-shot action recognition methods have achieved impressive results, they suffer from two major shortcomings. (a) During feature extraction, few-shot tasks are not distinguished and task-irrelevant features are obtained, resulting in the loss of task-specific critical discriminative information. (b) During feature matching, information critical to the features within the task, i.e., self-information and mutual information, is ignored, resulting in the accuracy being affected by redundant or irrelevant information. To overcome these two limitations, we propose a hierarchical task information mining (HiTIM) approach for few-shot action recognition that incorporates two key components: an inter-task learner (Kinter) and an attention-matching module with an intra-task learner (Kintra). The purpose of the Kinter is to learn the knowledge of different tasks and build a task-related feature space for obtaining task-specific features. The proposed matching module with Kintra consists of two branches: the spatiotemporal self-attention matching (STM) and correlated cross-attention matching (CM), which reinforce key spatiotemporal information in features and mine regions with strong correlations between features, respectively. The shared Kintra can further optimize STM and CM. In our method, we can use either a 2D convolutional neural network (CNN) or 3D CNN as embedding. In comparable experiments using two different embeddings in the five-way one-shot and five-way five-shot task, the proposed method achieved recognition accuracy that outperformed other state-of-the-art (SOTA) few-shot action recognition methods on the HMDB51 dataset and was comparable to SOTA few-shot action recognition methods on the UCF101 and Kinetics datasets.

Reconstructed Prototype Network Combined with CDC-TAGCN for Few-Shot Action Recognition

“…Although a layered structure improves efficiency, memory networks [17,18] consume significant storage space. As research progressed, relation networks [19], matching networks [20][21][22], and prototype networks [2,21,23] were gradually introduced. Jiang et al [19] used a hybrid relation module to capture relationships between different episodic tasks and utilized the mean hausdorff metric to measure the distance between query and support samples.…”

“…As research progressed, relation networks [19], matching networks [20][21][22], and prototype networks [2,21,23] were gradually introduced. Jiang et al [19] used a hybrid relation module to capture relationships between different episodic tasks and utilized the mean hausdorff metric to measure the distance between query and support samples. In one study by Careaga [21], an LSTM with a matching network framework was employed.…”