Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study

Abstract

Multi-Task Learning (MTL) has emerged as a methodology in which multiple tasks are jointly learned by a shared learning algorithm, such as a deep neural network. MTL is based on the assumption that the tasks under consideration are related; therefore it exploits shared knowledge for improving performance on each individual task. Tasks are generally considered to be homogeneous, i.e., to refer to the same type of problem, e.g., classification. Moreover, MTL is usually based on ground truth annotations with full, or partial overlap across tasks; i.e., for each input sample, there exist annotations for all or most of the tasks. In this work, we deal with heterogeneous MTL, simultaneously addressing detection, classification and regression problems. We explore task-relatedness as a means for co-training, in a weakly-supervised way, tasks that contain little, or even non-overlapping annotations. Task-relatedness is introduced in MTL, either explicitly through prior expert knowledge, or through data-driven studies. We propose a novel distribution matching approach, in which knowledge exchange is enabled between tasks, via matching of their predictions' distributions. Based on this approach, we build FaceBehaviorNet, the first framework for large-scale face analysis, by jointly learning all facial behavior tasks. We develop case studies for: i) continuous affect estimation, facial action unit detection and basic emotion recognition; ii) facial attribute detection and face identification. We illustrate that co-training via task relatedness alleviates negative transfer, i.e., cases in which MT model's performance is, in some task(s), worse than that of a single-task model. Since FaceBehaviorNet learns features that encapsulate all aspects of facial behavior, we conduct zero-and few-shot learning to perform tasks beyond the ones that it has been trained for, such as compound emotion recognition. By conducting a very large experimental study, utilizing 10 databases, we illustrate that our approach outperforms, by large margins, the state-of-the-art in all tasks and in all databases, even in these which have not been used in its training.

Prior Aided Streaming Network for Multi-task Affective Analysis

“…The 2nd Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW) [1,2,11,21,22,23,24] provides a benchmark dataset Aff-Wild2 for three recognition challenges: 7 basic emotion classification, 12 AUs detection and VA regression. Extened from Aff-wild [1], Aff-wild2 increases the number of annotated videos with 545 videos annotated by valence-arousal, 539 videos annotated by 7 basic emotion categories and 534 videos annotated by 12 AUs.…”

Causal affect prediction model using a facial image sequence

“…However, there are many obstacles for HCI systems used in real-world applications, such as in-the-wild or real-time tasks. To address these problems, Kollias et al have been hosting the Affective Behavior Analysis in-the-wild (ABAW) Competition, which involves a variety of research activites, for two years [5,6,7,8,9,11,12,28]. Most of the top-ranked teams in the first challenge of ABAW (ABAW1) [6], held in conjunction with the 15 th IEEE Conference on Face and Gesture Recognition (FG2020), used convolutional neural networks (CNNs) with single facial images or sequences of such images.…”

Action Units Recognition Using Improved Pairwise Deep Architecture

“…Affective Behavior Analysis in-the-wild (ABAW) including automatic AUs recognition competition was firstly held in FG2020 using Aff-wild2 database and this year is held in ICCV2021 [9,10,11,12,13,14,15,16,17,18,26]. In the competition, training and validation datasets that include multiple videos and AUs occurrence annotation for each frame image of the videos are provided.…”