Multiple Riemannian Manifold-Valued Descriptors Based Image Set Classification With Multi-Kernel Metric Learning

Abstract

The importance of wild video based image set recognition is becoming monotonically increasing due to the large amount of video resources obtained by diversified video collection approaches, like surveillance, drive recorders, smart phones, and internet. However, the contents of these collected videos are often complicated, and how to efficiently perform set modeling and feature extraction is a big challenge for set-based classification algorithms. In recent years, some proposed image set classification methods have made a considerable advance by modeling the original image set with covariance matrix, linear subspace, or Gaussian distribution. Moreover, the distinctive geometry spanned by them are Symmetric Positive Definite (SPD) manifold, Grassmann manifold, and Gaussian embedded Riemannian manifold, respectively. As a matter of fact, most of them just adopt a single geometric model to describe each given image set, which may lose some other useful information for classification. To tackle this problem, we propose a novel algorithm to model each image set from a multi-geometric perspective. Specifically, the covariance matrix, linear subspace, and Gaussian distribution are applied for set representation simultaneously. In order to fuse these multiple heterogeneous Riemannian manifoldvalued features, the well-equipped Riemannian kernel functions are first utilized to map them into high dimensional Hilbert spaces. Then, a multi-kernel metric learning framework is devised to embed the learned hybrid kernels into a lower dimensional common subspace for classification. We conduct experiments on four widely used datasets corresponding to four different classification tasks: video-based face recognition, set-based object categorization, video-based emotion recognition, and dynamic scene classification, to evaluate the classification performance of the proposed algorithm. Extensive experimental results justify its superiority over the state-of-the-art.

DreamNet: A Deep Riemannian Manifold Network for SPD Matrix Learning

“…In this paper, several representative Riemannian manifold learning-based visual classification methods are selected to better evaluate the effectiveness of the proposed approach, which can be grouped into the following four categories: 1) general methods for SPD matrix learning, including Log-Euclidean Metric Learning (LEML) [10] and SPD Manifold Learning (SPDML) [12]; 2) general methods for linear subspace learning, including Projection Metric Learning (PML) [38], Graph Embedding Projection Metric Learning (GEPML) [39], and Graph Embedding Multi-Kernel Metric Learning (GEMKML) [40]; 3) multi-order statistical learning methods, containing Hybrid Euclidean-and-Riemannian Metric Learning (HERML) [11] and Multiple Riemannian Manifolds Metric Learning (MRMML) [41]; 4) Riemannian deep learning methods, containing Grassmannian Neural Network (GrNet) [42], SPD Neural Network (SPDNet) [7], SPDNet embedded with Riemannian Batch Normalization (SPDNetBN) [8], Lightweight SPD Neural Network (SymNet) [5], Manifoldvalued Deep Network (ManifoldNet) [3], and our baseline model, i.e., Deep SPDNet (DSPDNet) [6].…”

Unified Tensor Framework for Incomplete Multi-view Clustering and Missing-view Inferring

“…In fields of clustering, graph learning based methods are popular and have received lots of attention in recent years (Nie et al 2016;Wan and Meila 2016;Wang, Wu, and Kittler 2020;Wang et al 2020b;Xie et al 2020). For a given set of data points, it seeks to construct the intrinsic graph whose every element represents a kind of similarity of the corresponding two points for clustering.…”

MSNet: A Deep Multi-scale Submanifold Network for Visual Classification

“…Some of these approaches attempt to learn a tangent map via logarithms of SPD matrices [21], or map the original SPD matrices into Reproducing Kernel Hilbert Spaces (RKHS) [20], [29], [30], [3]. However, these methods tend to distort the geometrical structure communicated by the data, since they model the manifold indirectly.…”