Deep Fusion: An Attention Guided Factorized Bilinear Pooling for Audio-video Emotion Recognition

Abstract

Automatic emotion recognition (AER) is a challenging task due to the abstract concept and multiple expressions of emotion. Although there is no consensus on a definition, human emotional states usually can be apperceived by auditory and visual systems. Inspired by this cognitive process in human beings, it's natural to simultaneously utilize audio and visual information in AER. However, most traditional fusion approaches only build a linear paradigm, such as feature concatenation and multi-system fusion, which hardly captures complex association between audio and video. In this paper, we introduce factorized bilinear pooling (FBP) to deeply integrate the features of audio and video. Specifically, the features are selected through the embedded attention mechanism from respective modalities to obtain the emotion-related regions. The whole pipeline can be completed in a neural network. Validated on the AFEW database of the audio-video sub-challenge in EmotiW2018, the proposed approach achieves an accuracy of 62.48%, outperforming the state-of-the-art result.

Hand-Gesture Recognition Based on EMG and Event-Based Camera Sensor Fusion: A Benchmark in Neuromorphic Computing

“…The higher accuracy is achieved thanks to a redundancy gain that reduces the amount of uncertainty in the resulting information. Recent works show a growing interest toward multi-sensory fusion in several application areas, such as developmental robotics (Droniou et al, 2015 ; Zahra and Navarro-Alarcon, 2019 ), audio-visual signal processing (Shivappa et al, 2010 ; Rivet et al, 2014 ), spatial perception (Pitti et al, 2012 ), attention-driven selection (Braun et al, 2019 ) and tracking (Zhao and Zeng, 2019 ), memory encoding (Tan et al, 2019 ), emotion recognition (Zhang et al, 2019 ), multi-sensory classification (Cholet et al, 2019 ), HMI (Turk, 2014 ), remote sensing and earth observation (Debes et al, 2014 ), medical diagnosis (Hoeks et al, 2011 ), and understanding brain functionality (Horwitz and Poeppel, 2002 ).…”

Brain-Inspired Self-Organization with Cellular Neuromorphic Computing for Multimodal Unsupervised Learning

“…• First, there is an independent activity computation (Equation (13)): each neuron of the two SOMs computes its activity based on the afferent activity from the input. • Second, there is a cooperation amongst neurons from different modalities (Equations (14) and (15)): each neuron updates its afferent activity via a multiplication with the lateral activity from the neurons of the other modality. • Third and finally, there is a global competition amongst all neurons (line 19 in Algorithm 4): they all compete to elect a winner, that is, a global BMU with respect to the two SOMs.…”

“…Multimodal data fusion is thus a direct consequence of the well-accepted paradigm that certain natural processes and phenomena are expressed under completely different physical guises [6]. Recent works show a growing interest toward multimodal association in several applicative areas such as developmental robotics [3], audio-visual signal processing [7,8], spatial perception [9,10], attention-driven selection [11] and tracking [12], memory encoding [13], emotion recognition [14], human-machine interaction [15], remote sensing and earth observation [16], medical diagnosis [17], understanding brain functionality [18], and so forth. Interestingly, the last mentioned application is our starting bloc: how does the brain handle multimodal learning in the natural environment?…”

Exploring Emotion Features and Fusion Strategies for Audio-Video Emotion Recognition

“…Audio processing and Spectrogram calculation. For each audio, the speech spectrogram and log Mel-spectrogram extraction process is consistent with [32] and [3] respectively. For speech spectrogram, we use the Hamming window with 40 msec window size and 10 msec shift.…”