Multi-label zero-shot human action recognition via joint latent ranking embedding

Abstract

Human action recognition refers to automatic recognizing human actions from a video clip, which is one of the most challenging tasks in computer vision. Due to the fact that annotating video data is laborious and time-consuming, most of the existing works in human action recognition are limited to a number of small scale benchmark datasets where there are a small number of video clips associated with only a few human actions and a video clip often contains only a single action. In reality, however, there often exist multiple human actions in a video stream. Such a video stream is often weakly-annotated with a set of relevant human action labels at a global level rather than assigning each label to a specific video episode corresponding to a single action, which leads to a multi-label learning problem. Furthermore, there are a great number of meaningful human actions in reality but it would be extremely difficult, if not impossible, to collect/annotate video clips regarding all of various human actions, which leads to a zero-shot learning scenario. To the best of our knowledge, there is no work that has addressed all the above issues together in human action recognition. In this paper, we formulate a real-world human action recognition task as a multi-label zero-shot learning problem and propose a framework to tackle this problem in a holistic way. Our framework holistically tackles the issue of unknown temporal boundaries between different actions for multi-label learning and exploits the side information regarding the semantic relationship between different human actions for knowledge transfer. As a result, our framework leads to a joint latent ranking embedding for multi-label zero-shot human action recognition. A novel neural architecture of two component models and an alternate learning algorithm are proposed to carry out the joint latent ranking embedding learning. Thus, multi-label zero-shot recognition is done by measuring relatedness scores of action labels to a test video clip in the joint latent visual and semantic embedding spaces. We evaluate our framework with different settings, including a novel data split scheme designed especially for evaluating multi-label zero-shot learning, on two weakly annotated multi-label human action datasets: Breakfast and Charades. The experimental results demonstrate the effectiveness of our framework in multi-label zero-shot human action recognition.

A Review of Generalized Zero-Shot Learning Methods

“…Human action/gesture recognition from both images and videos is one of the most challenging tasks in computer vision, due to a variety of actions that are not available among the seen action categories. In this regard, GZSLbased frameworks have been employed to recognize single label [24], [135], [159] and multi-label [158] human actions. As an example, a new multi-label ZSL (MZSL) framework using JLRE (Joint Latent Ranking Embedding) has been proposed in [158].…”

“…In this regard, GZSLbased frameworks have been employed to recognize single label [24], [135], [159] and multi-label [158] human actions. As an example, a new multi-label ZSL (MZSL) framework using JLRE (Joint Latent Ranking Embedding) has been proposed in [158]. The relatedness score of various action labels is measured for the test video clips in the semantic embedding and joint latent visual spaces.In addition, a multimodal framework using audio, video, and text has been introduced in [160], [161].…”

S3GRN: Structural Similar Stepwise Generative Recognizable Network for Human Action Recognition With Limited Training Data

“…are the standard deviation. 1 C , 2 C and 3 C are constants. The structure similarity function of the images are generated by combining the three variables:…”

“…Human action recognition is a hot topic in computer vision and pattern recognition [1], which has been applied to various fields, such as medical health [2], intelligent space [3], interactive entertainment [4], robotics [5] and so on. The action information can be obtained from medical equipment to help patients with rehabilitation training.…”

A Baseline for Multi-Label Image Classification Using an Ensemble of Deep Convolutional Neural Networks

“…Multi-label image classification has been a hot topic in computer vision community. Its extensive applications include but are not limited to image retrieval, automatic image annotation, web image search and image tagging [1,2,3,4,5].…”