Multiple Kernel Learning and Optical Flow for Action Recognition in RGB-D Video

Abstract

Recognizing human action is valuable for many real world applications such as video surveillance, human computer interaction, smart home and gaming. In this paper, we present a method of action recognition based on hypothesizing that the classification of action can be boosted by motion information using optical flow. Emergence of automatic RGBD video analysis, we propose fusing optical flow is extracted from both RGB and depth channels for action representation. Firstly, we extract optical flow from RGB and depth data. Secondly, motion descriptor with spatial pyramid is computed from histogram of optical flow of RGB and depth. Then, feature pooling technique is used in order to accumulate RGB and depth feature into set of feature vectors for each action. Finally, we use the Multiple Kernel Learning (MKL) technique at the kernel level for action classification from RGB and depth feature pooling. To demonstrate generalizability, our proposed method has been systematically evaluated on two benchmark datasets shown to be more effective and accurate for action recognition compared to the previous work. We obtain overall accuracies of: 97.5 % and 92.8 % with our proposed method on the 3D ActionPairs and MSR-Daily Activity 3D dataset, respectively.

Combining Pose-Invariant Kinematic Features and Object Context Features for RGB-D Action Recognition

“…A fundamental aspect of any action is the articulated motion of one or more body parts. To capture motion, [6] extracted optical flow from RGB and depth data and represented actions using spatial pyramid histogram of optical flow. Meanwhile, [7] introduced a depth motion map, that accumulates foreground motion re-gions to capture global activities.…”

“…We can observe that the proposed method performs better than most of the available methods. Heterogeneous feature learning [17] and multiple kernel learning [6] allow them to learn significantly important motion and appearance in an action, thus allowing for a better action representation. The proposed framework is pose-invariant whereas the other methods available are quite limited.…”

“…Since the skeleton label is not provided for depth and RGB frames, we include a foreground detection based on the x,y and z coordinates of the skeleton. [24] 95.63 Heterogeneous Learning [17] 95 Multiple Kernel Learning [6] 92.5 Actionlets [16] 85.75 Tensor Subspace [25] 80.63…”