Jointly Learning Heterogeneous Features for RGB-D Activity Recognition

Abstract

In this paper, we focus on heterogeneous features learning for RGB-D activity recognition. We find that features from different channels (RGB, depth) could share some similar hidden structures, and then propose a joint learning model to simultaneously explore the shared and feature-specific components as an instance of heterogeneous multi-task learning. The proposed model formed in a unified framework is capable of: 1) jointly mining a set of subspaces with the same dimensionality to exploit latent shared features across different feature channels, 2) meanwhile, quantifying the shared and feature-specific components of features in the subspaces, and 3) transferring feature-specific intermediate transforms (i-transforms) for learning fusion of heterogeneous features across datasets. To efficiently train the joint model, a three-step iterative optimization algorithm is proposed, followed by a simple inference model. Extensive experimental results on four activity datasets have demonstrated the efficacy of the proposed method. A new RGB-D activity dataset focusing on human-object interaction is further contributed, which presents more challenges for RGB-D activity benchmarking.

Deep-Aligned Convolutional Neural Network for Skeleton-Based Action Recognition and Segmentation

“…SYSU dataset: For the empirical evaluations, we compare our DACNN algorithm to other baselines including CNN+DPRL [28], ST-LSTM+Trust Gate [13], Dynamic Skeletons [35], LAFF(SKL) [45], SR-TSL [46], VA-LSTM [47], and GCA-LSTM [48], which includes the most recent deep learning applications (CNN, LSTM, etc.) on this dataset.…”

“…SYSU-3D Human-Object Interaction dataset (SYSU) [35]: This dataset contains 12 different action classes recorded in 480 video sequences. SYSU dataset is captured from 40 human subjects, and in each time-frame, it represents the 3D coordinates of 20 body joints.…”

Deep spatiotemporal LSTM network with temporal pattern feature for 3D human action recognition

“…An intuitive way to combine multimodal features is to directly concatenate them together . To mine more useful information among multimodal features for better performance, researchers propose to explicitly learn shared‐specific structures among features …”

“…52 To mine more useful information among multimodal features for better performance, researchers propose to explicitly learn shared-specific structures among features. 11,53 Early on, Liu and Shao 54 utilized a genetic programming framework to improve not only RGB and depth descriptors but also their fusion simultaneously through an iterative evolution. Ni et al 55 concatenated depth descriptor-and RGB-based representations of spatiotemporal interest points for better RGB+D information fusion.…”

“…Generally speaking, there are two kinds of methods for validation in action recognition, ie, leave-one-out cross validation and 2-fold cross validation. We follow the idea in the works of Liu et al, 13 Xia et al, 33 and Hu et al 53 to utilize leave-one-out cross validation for the experiments. Florence 3D data set collects data through a stationary Kinect as well, collecting nine common indoor action categories, such as "watching," "drinking water," "calling," and so on.…”