Regularized Chained Deep Neural Network Classifier for Multiple Annotators

Abstract

The increasing popularity of crowdsourcing platforms, i.e., Amazon Mechanical Turk, changes how datasets for supervised learning are built. In these cases, instead of having datasets labeled by one source (which is supposed to be an expert who provided the absolute gold standard), databases holding multiple annotators are provided. However, most state-of-the-art methods devoted to learning from multiple experts assume that the labeler’s behavior is homogeneous across the input feature space. Besides, independence constraints are imposed on annotators’ outputs. This paper presents a regularized chained deep neural network to deal with classification tasks from multiple annotators. The introduced method, termed RCDNN, jointly predicts the ground truth label and the annotators’ performance from input space samples. In turn, RCDNN codes interdependencies among the experts by analyzing the layers’ weights and includes l1, l2, and Monte-Carlo Dropout-based regularizers to deal with the over-fitting issue in deep learning models. Obtained results (using both simulated and real-world annotators) demonstrate that RCDNN can deal with multi-labelers scenarios for classification tasks, defeating state-of-the-art techniques.

Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification

“…Yet, while such approaches code the annotators’ parameters as fixed points, we model them as functions to consider dependencies between the input features and the labelers’ behavior. GCECDL is also similar to the works in [ 14 , 43 ]. Both approaches model the annotators’ performance as a function of the input instances and consider the interdependencies among the labelers.…”

“…This article introduced a Generalized Cross-Entropy-based Chained Deep Learning model, termed GCECDL, to deal with multiple-annotator scenarios. Our method follows the ideas of [ 43 , 46 ], where each parameter is modeled in a multi-labeler likelihood by using the outputs of a deep neural network. Nonetheless, unlike [ 43 ]—where a CCE-based loss was used—we also introduced a noise-robust loss function based on GCE [ 42 ] as a tradeoff between MAE and CCE.…”

“…Even so, unlike [ 14 ], where it is necessary to use as many classifiers as annotators, our approach only needs to train a single classifier from a DL representation, which is advantageous for a large number of labelers. Moreover, unlike [ 43 ], our loss function can deal with noisy labels and more difficult training scenarios by using a generalization between MAE and CE. Indeed, network self-regularization is accomplished by predicting each labeler’s reliability.…”

“…Moreover, Table 2 displays the tested state-of-the-art algorithms for comparison purposes. The abbreviations are fixed as follows: Regularized Chained Deep Neural Network Classifier for Multiple Annotators (RCDNN) [ 43 ], Deep Learning Majority Voting (DL-MV), and Deep Learning from Crowds (DL-CW(MW)) [ 28 ]. Our Python codes are publicly available for DL-CL(MW), DL-MV, RCDNN, and GCECDL at (accessed on 19 December 2022).…”

Random Fourier Features-Based Deep Learning Improvement with Class Activation Interpretability for Nerve Structure Segmentation

“…In addition, a non-parametric Friedman test was computed for statistical significance. The null hypothesis was that all algorithms perform equally [54,55]. For concrete testing, we fixed the significance threshold as p-value < 0.05.…”