A Classification Model for Diverse and Noisy Labelers

Sung, Hao-En; Chen, Cheng-Kuan; Xiao, Han; Lin, Shou-De

doi:10.1007/978-3-319-57454-7_5

Cited by 2 publications

(1 citation statement)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the ML field, assigning labels to instances with the help of multiple annotators is a common practice. However, it presents a significant challenge when traditional supervised algorithms are applied because they rely on the assumption that the training labels provided by a single expert are reliable [ 5 ]. When multiple annotators with varying levels of expertise are employed, the reliability of the labels becomes uncertain, leading to decreased performance and inaccurate model predictions.…”

Section: Introductionmentioning

confidence: 99%

Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification

Triana-Martinez

Gil-Gonzalez

Fernandez-Gallego

et al. 2023

Sensors

View full text Add to dashboard Cite

Supervised learning requires the accurate labeling of instances, usually provided by an expert. Crowdsourcing platforms offer a practical and cost-effective alternative for large datasets when individual annotation is impractical. In addition, these platforms gather labels from multiple labelers. Still, traditional multiple-annotator methods must account for the varying levels of expertise and the noise introduced by unreliable outputs, resulting in decreased performance. In addition, they assume a homogeneous behavior of the labelers across the input feature space, and independence constraints are imposed on outputs. We propose a Generalized Cross-Entropy-based framework using Chained Deep Learning (GCECDL) to code each annotator’s non-stationary patterns regarding the input space while preserving the inter-dependencies among experts through a chained deep learning approach. Experimental results devoted to multiple-annotator classification tasks on several well-known datasets demonstrate that our GCECDL can achieve robust predictive properties, outperforming state-of-the-art algorithms by combining the power of deep learning with a noise-robust loss function to deal with noisy labels. Moreover, network self-regularization is achieved by estimating each labeler’s reliability within the chained approach. Lastly, visual inspection and relevance analysis experiments are conducted to reveal the non-stationary coding of our method. In a nutshell, GCEDL weights reliable labelers as a function of each input sample and achieves suitable discrimination performance with preserved interpretability regarding each annotator’s trustworthiness estimation.

show abstract

Section: Introductionmentioning

confidence: 99%

Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification

Triana-Martinez

Gil-Gonzalez

Fernandez-Gallego

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

Regularized Chained Deep Neural Network Classifier for Multiple Annotators

et al. 2021

View full text Add to dashboard Cite

The increasing popularity of crowdsourcing platforms, i.e., Amazon Mechanical Turk, changes how datasets for supervised learning are built. In these cases, instead of having datasets labeled by one source (which is supposed to be an expert who provided the absolute gold standard), databases holding multiple annotators are provided. However, most state-of-the-art methods devoted to learning from multiple experts assume that the labeler’s behavior is homogeneous across the input feature space. Besides, independence constraints are imposed on annotators’ outputs. This paper presents a regularized chained deep neural network to deal with classification tasks from multiple annotators. The introduced method, termed RCDNN, jointly predicts the ground truth label and the annotators’ performance from input space samples. In turn, RCDNN codes interdependencies among the experts by analyzing the layers’ weights and includes l1, l2, and Monte-Carlo Dropout-based regularizers to deal with the over-fitting issue in deep learning models. Obtained results (using both simulated and real-world annotators) demonstrate that RCDNN can deal with multi-labelers scenarios for classification tasks, defeating state-of-the-art techniques.

show abstract

A Classification Model for Diverse and Noisy Labelers

Cited by 2 publications

References 9 publications

Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification

Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification

Regularized Chained Deep Neural Network Classifier for Multiple Annotators

Contact Info

Product

Resources

About