Sparse Coding of Shape Trajectories for Facial Expression and Action Recognition

Abstract

The detection and tracking of human landmarks in video streams has gained in reliability partly due to the availability of affordable RGB-D sensors. The analysis of such time-varying geometric data is playing an important role in the automatic human behavior understanding. However, suitable shape representations as well as their temporal evolution, termed trajectories, often lie to nonlinear manifolds. This puts an additional constraint (i.e., nonlinearity) in using conventional Machine Learning techniques. As a solution, this paper accommodates the well-known Sparse Coding and Dictionary Learning approach to study time-varying shapes on the Kendall shape spaces of 2D and 3D landmarks. We illustrate effective coding of 3D skeletal sequences for action recognition and 2D facial landmark sequences for macro-and micro-expression recognition. To overcome the inherent nonlinearity of the shape spaces, intrinsic and extrinsic solutions were explored. As main results, shape trajectories give rise to more discriminative time-series with suitable computational properties, including sparsity and vector space structure. Extensive experiments conducted on commonly-used datasets demonstrate the competitiveness of the proposed approaches with respect to state-of-the-art.

3D Skeletal Gesture Recognition via Discriminative Coding on Time-Warping Invariant Riemannian Trajectories

“…In comparison to [46], which used the fixed point for embedding, [48] operated on the tangent bundle, meaning that each point of the manifold was coded on its attached tangent space where the atoms are mapped. Following [48], Tanfous et al [49] explored sparse coding and dictionary learning in the Kendall shape spaces (Kendall-SCDL), aiming to study the time-varying shapes of 3D skeleton trajectories for action recognition. However, Kendall-SCDL [49] has a mandatory step of dictionary initialization that heavily relies on principal geodesic analysis (PGA) [50] to generate atoms.…”

“…Following [48], Tanfous et al [49] explored sparse coding and dictionary learning in the Kendall shape spaces (Kendall-SCDL), aiming to study the time-varying shapes of 3D skeleton trajectories for action recognition. However, Kendall-SCDL [49] has a mandatory step of dictionary initialization that heavily relies on principal geodesic analysis (PGA) [50] to generate atoms. Also, Kendall-SCDL is still an unsupervised model.…”

“…The first group is a group of the methods most related to ours, including five Lie group representation-based algorithms: Lie group using DTW [4] (Lie group-DTW), Lie group with TSRVF [45] (Lie group-TSRVF), Lie group with TSRVF and using PCA for dimensionality reduction [44] (Lie group-TSRVF-PCA), Lie group with TSRVF and K-SVD for sparse coding [59] (Lie group-TSRVF-KSVD), and the Lie group with deep learning (LieNet) [51], as well as two TSRVFrelated methods, the body part features with SRV and knearest neighbors clustering [47] (SRV-KNN), and TSRVF on Kendall's shape [5] (Kendall-TSRVF). In addition, two recent manifold-based methods, namely the Kendall-SCDL [49] and Gramian matrices [54], are compared. The methods in the second group are based on classic feature representations, like HOJ3D [12], EigenJoints [13], actionlet ensemble (Actionlet) [17], HON4D [15], discriminative key-frames (Key-frames) [14], RVV with DTW (RVV-DTW) [10], and spatio-temporal naive Bayes nearest-neighbor (ST-NBNN) [25].…”

“…We adopt the evaluation protocol provided by the dataset which assigns 7 754 gesture sequences for training, 3 362 sequences for validation, and 2 742 sequences for testing. To the best of our knowledge, ChaLearn 2014 is one of Methods Accuracy Lie group-DTW [4] 92.5 Lie group-TSRVF [45] 87.7 Lie group-TSRVF-PCA [44] 88.3 Lie group-TSRVF-KSVD [59] 87.6 SRV-KNN [47] 92.1 Kendall-TSRVF [5] 89.9 Kendall-SCDL [49] 94.2 EigenJoints [13] 82.3 Actionlet [17]* 88.2 HOJ3D [12] 78.9 HON4D [15]* 88.9 Key-frames [14] 91.7 RVV-DTW [10] 93.4 ST-NBNN [25] 94.8…”

“…In fact, the ST-NBNN is based on naive-Bayes nearest-neighbor (NBNN) distance matrices, and a specifically-designed tensor SVM is introduced to improve classification accuracy. However, Methods Accuracy Lie group-DTW [4] 97.1 Lie group-TSRVF [45] 94.5 Lie group-TSRVF-PCA [44] 94.9 Lie group-TSRVF-KSVD [59] 92.7 SRV-KNN [47] 91.5 Kendall-TSRVF [5] 89.8 Gramian matrix [54] 96.5 Kendall-SCDL [49] 97.5 EigenJoints [13] 92.4 HOJ3D [12] 90.9 HON4D [15]* 90.9 ST-NBNN [25] 98.0 as mentioned when discussing the experimental settings, we only employ the linear SVM for classification, as with many comparative methods [4] [5] [44] [45] [59]. Actually, as reported in [25], with the same setting namely the NBNN with weight learning using linear SVM (NBNN+SVM) yields a worse score (92.4%) than ours.…”