Joint patch and multi-label learning for facial action unit detection

Abstract

The face is one of the most powerful channel of nonverbal communication. The most commonly used taxonomy to describe facial behaviour is the Facial Action Coding System (FACS). FACS segments the visible effects of facial muscle activation into 30+ action units (AUs). AUs, which may occur alone and in thousands of combinations, can describe nearly all-possible facial expressions. Most existing methods for automatic AU detection treat the problem using one-vs-all classifiers and fail to exploit dependencies among AU and facial features. We introduce joint-patch and multi-label learning (JPML) to address these issues. JPML leverages group sparsity by selecting a sparse subset of facial patches while learning a multi-label classifier. In four of five comparisons on three diverse datasets, CK+, GFT, and BP4D, JPML produced the highest average F1 scores in comparison with state-of-the art.

Joint Facial Action Unit Detection and Feature Fusion: A Multi-Conditional Learning Approach

“…Yet, the complexity of the model in [17] increases quadratically with the number of AUs in the output, while it increases only linearly in case of MC-LVM. Consequently, MC-LVM can efficiently model relations among a relatively large number of outputs, without the requirement to a priori define groups of AUs as done in [17], [14].…”

“…Hence, S ij contains the number of co-activated outputs in all sub-labels between two instances. Note that contrary to [14], we measure the similarity of the outputs based on all possible groups of co-occurring AUs, and not only on pairs of AUs. The graph Laplacian matrix is then defined as L = D − S, where D is a diagonal matrix with D ii = j S ij .…”

“…We also compare to the multi-label backpropagation and kNN (k=1), i.e., the BPMLL [39] and ML-KNN [38], respectively. Lastly, we compare to the state-of-the-art methods for multiple AU detection: the parametric methods Bayesian groupsparse compressed sensing (BGCS) [13], hierarchical RBM (HRBM) [15], joint patch multi-label learning (JPML) [14], and the kernel method l p -regularized multi-task MKL (l p -MTMKL) [17]. All the compared methods are evaluated using the same previously described input features.…”

“…Note that this model fails to account for AU dependencies at the feature level, which can result in loss of information, e.g., in case of non-additive AUs. [14] tries to learn independent logistic classifiers by first selecting a sparse subset of facial patches which are more relevant to each AU. Yet, the fusion task is not addressed, while the AU-dependencies are regarded only between predefined pairs.…”

“…Furthermore, some methods treat different combinations of AUs as new independent classes [11]; however, this is impractical given the number of possible AU combinations. On the other hand, methods that do attempt to model the AU co-occurrences (e.g., [12], [13], [14]) fail to exploit different types of facial features in their models. To our knowledge, the only methods that attempt both are [15], [16], [17].…”