Neural conditional ordinal random fields for agreement level estimation

Abstract

Abstract-We present a novel approach to automated estimation of agreement intensity levels from facial images. To this end, we employ the MAHNOB Mimicry database of subjects recorded during dyadic interactions, where the facial images are annotated in terms of agreement intensity levels using the Likert scale (strong disagreement, disagreement, neutral, agreement and strong agreement). Dynamic modelling of the agreement levels is accomplished by means of a Conditional Ordinal Random Field model. Specifically, we propose a novel Neural Conditional Ordinal Random Field model that performs non-linear feature extraction from face images using the notion of Neural Networks, while also modelling temporal and ordinal relationships between the agreement levels. We show in our experiments that the proposed approach outperforms existing methods for modelling of sequential data. The preliminary results obtained on five subjects demonstrate that the intensity of agreement can successfully be estimated from facial images (39% F1 score) using the proposed method.

Multi-modal Neural Conditional Ordinal Random Fields for agreement level estimation

“…Other lines of work deal with the analysis of (dis)agreement expression channels and verbal/non-verbal cues [6], [1], estimation based on lexical and text-based data [7], as well as audio and prosody cues [8]. Due to the variety of means by which the (dis)agreement can be communicated, we adopt the multi-level annotation scale introduced in [9]. The agreement levels are represented using Likert scale [10], where the intensity of agreement ranges from strong disagreement to strong agreement.…”

“…The agreement levels are represented using Likert scale [10], where the intensity of agreement ranges from strong disagreement to strong agreement. In particular, the (dis)agreement levels are defined as: neutral {0}, (dis)agreement {-1,+1}, strong (dis)agreement {-2,+2} as defined in [9].…”

“…However, these methods fail to account for ordinal information in the target data. We extend the standard CORF model [21], and uni-modal Neural CORF [9] to design the multi-modal NCORF approach for the target task. In all these models, we first define the conditional distribution P (Y|X) of having a label sequence Y, based on the observation sequence X, as:…”

“…To train and evaluate the performance of the model we used the MAHNOB-Mimicry database [26] for which we performed the agreement intensity level annotation, as in [9]. The database consists of video recordings of 54 dyadic discussion sessions.…”

“…The database consists of video recordings of 54 dyadic discussion sessions. We selected videos of 38 subjects, an extension of [9], where authors used only 5 subjects. Labelling was done using segments, which could be defined as a generalisation of 'spurts' -periods of speech by one speaker that have no pauses greater than 0.5 second (similarly as defined in [27]) -where both audio and visual modality were considered.…”