Latent Multitask Learning for View-Invariant Action Recognition

Abstract

This paper presents an approach to view-invariant action recognition, where human poses and motions exhibit large variations across different camera viewpoints. When each viewpoint of a given set of action classes is specified as a learning task then multitask learning appears suitable for achieving view invariance in recognition. We extend the standard multitask learning to allow identifying: (1) latent groupings of action views (i.e., tasks), and (2) discriminative action parts, along with joint learning of all tasks. This is because it seems reasonable to expect that certain distinct views are more correlated than some others, and thus identifying correlated views could improve recognition. Also, part-based modeling is expected to improve robustness against self-occlusion when actors are imaged from different views. Results on the benchmark datasets show that we outperform standard multitask learning by 21.9%, and the state-of-the-art alternatives by 4.5-6%.

Multitask Linear Discriminant Analysis for View Invariant Action Recognition

“…To our knowledge, multi-view action recognition using multi-task learning has not been considered by other works with the exception of [21], which is contemporaneous to ours. Also, our approach is different with respect to [21] in the following respects: (1) While [21] seeks to learn latent action groups, so that within-group feature sharing is allowed but between-group feature sharing is prohibited, we explore learning of latent and discriminative SSM features across views; (2) A part-based action representation is used in [21], while we use the bag-of-words model for encoding SSM features; (3) A large-margin framework is used for LMTL formulation in [21], while we propose LDA-based MTL, and (4) While in [21] the main focus is multi-view action recognition, we also consider the problem of action recognition with missing view, i.e. on a novel camera view for which no examples are available in the training set.…”

“…Our approach achieves higher recognition performance, in terms of both single-view and (average) multi-view accuracies, as compared to most previous methods. While the approaches proposed in [16], [21], and [55] achieve higher recognition as compared to our methods, they suffer from other limitations. The algorithm in [55] is based on latent kernelized structural SVM which is intractable for inference on largescale datasets.…”

“…Differently, our method is computationally efficient and also easy to implement. It is particularly interesting to compare our approach with [21], as this is also based on multi-task learning. The best performance reported in [21] is achieved by combining MTL with a feature representation which takes into account the layout of body parts.…”

“…It is particularly interesting to compare our approach with [21], as this is also based on multi-task learning. The best performance reported in [21] is achieved by combining MTL with a feature representation which takes into account the layout of body parts. This representation is very powerful and generally superior to the bag-of-words approach, which we adopt in this paper.…”

“…While other recent methods such as [21] also adopt MTL, a unique aspect of our framework is that, by effectively combining SSMs descriptors, sparsity and MTL, it easily allows for accurate action classification on missing views, for which no examples are available in the training data.…”