Modeling Multiple Time Series Annotations as Noisy Distortions of the Ground Truth: An Expectation-Maximization Approach

Abstract

Studies of time-continuous human behavioral phenomena often rely on ratings from multiple annotators. Since the ground truth of the target construct is often latent, the standard practice is to use ad-hoc metrics (such as averaging annotator ratings). Despite being easy to compute, such metrics may not provide accurate representations of the underlying construct. In this paper, we present a novel method for modeling multiple time series annotations over a continuous variable that computes the ground truth by modeling annotator specific distortions. We condition the ground truth on a set of features extracted from the data and further assume that the annotators provide their ratings as modification of the ground truth, with each annotator having specific distortion tendencies. We train the model using an Expectation-Maximization based algorithm and evaluate it on a study involving natural interaction between a child and a psychologist, to predict confidence ratings of the children’s smiles. We compare and analyze the model against two baselines where: (i) the ground truth in considered to be framewise mean of ratings from various annotators and, (ii) each annotator is assumed to bear a distinct time delay in annotation and their annotations are aligned before computing the framewise mean.

Generating labels for regression of subjective constructs using triplet embeddings

“…We compare the proposed annotation and fusion framework with two different approaches using real-time annotations: EvalDep [15] and the EM-based approach (after time-alignment using EvalDep's method) from [10] with window lengths of 4, 8, 16, and 32. Fig.…”

“…[23] proposes the use of a Long-Short-Term-Memory network (LSTM) to fuse asynchronous input annotations, by conducting time-alignment and de-biasing the different annotations. [10] presents a method for modeling multiple annotations over a continuous variable, and computes the ground truth by modeling annotator-specific distortions as filters whose parameters can be estimated jointly using Expectation-Maximization (EM). However, this work relies on heavy assumptions in the models for mathematical tractability, that do not necessarily reflect how annotators behave.…”

Labeler-hot Detection of EEG Epileptic Transients

“…Recently, to overcome high redundancy constraint, an EM algorithm used predicated label as ground truth to estimate labeler confusion matrix [16]. There are also results specifically in the area of time series labeling, which are more related to EEG annotations than image labeling [17], [18].…”

“…Another line of works tweaks loss function to incorporate assumption about uniform noise process disturbing labels [19], [16], [17]. There was significant amount of work in the area of active learning [20], [21] that ask for more labels of inconsistent examples.…”

Towards a Better Gold Standard

“…Only a few studies have explored the pre-processing and postprocessing of the distribution of arousal/valence annotations [18,41,28]. There are a few papers working on rating annotators or maximising the mutual information of multiple annotations [5,19,18]. However, so far there is no study which compares the performance of several denoising and dimensionality reduction methods.…”