Convex Formulations of Learning from Crowds

Abstract

It has attracted considerable attention to use crowdsourcing services to collect a large amount of labeled data for machine learning, since crowdsourcing services allow one to ask the general public to label data at very low cost through the Internet. The use of crowdsourcing has introduced a new challenge in machine learning, that is, coping with low quality of crowd-generated data. There have been many recent attempts to address the quality problem of multiple labelers, however, there are two serious drawbacks in the existing approaches, that are, (i) non-convexity and (ii) task homogeneity. Most of the existing methods consider true labels as latent variables, which results in nonconvex optimization problems. Also, the existing models assume only single homogeneous tasks, while in realistic situations, clients can offer multiple tasks to crowds and crowd workers can work on different tasks in parallel. In this paper, we propose a convex optimization formulation of learning from crowds by introducing personal models of individual crowds without estimating true labels. We further extend the proposed model to multi-task learning based on the resemblance between the proposed formulation and that for an existing multi-task learning model. We also devise efficient iterative methods for solving the convex optimization problems by exploiting conditional independence structures in multiple classifiers.

A Multi-task Learning Framework for Time-continuous Emotion Estimation from Crowd Annotations

“…The intuition behind MTL is simple: a joint learning procedure which accounts for task relationships is expected to lead to more accurate models as compared to learning each task separately. While MTL has been used previously for learning from noisy crowd annotations [9], we present the first work that employs MTL for time-continuous emotion prediction from movie clips.…”

“…While MTL has been used to learn from noisy crowd data [9], we simply used the median value of the annotations at each time-point to derive the ground truth emotional profile for each movie clip. Next, we will briefly describe the audio-visual features extracted from each clip, and show how the joint learning of the relationship between audiovisual features and VA ratings allows for more effective dynamic emotion prediction.…”

Recovering Missing Labels of Crowdsourcing Workers

“…Another observation of this experiment is the fast convergence speed of the iterative algorithm for inferring the Finally we show the power of our recovered labels in predicting the ground truth. For baseline we use the Personal Classifier (PC) model [8] and Latent Class (LC) model [16], two state of art models for learning the true classifier directly. We use the baseline models to learn classifiers from both the original incomplete label set and full label set recovered by our CC method.…”

Learning from Crowds under Experts’ Supervision

“…Thus in crowdsourcing, people may collect multiple labels y The problem remains as how to learn a reliable predictive model with the unreliable crowd labels. Various methods have been proposed to infer the ground truth [4,10] or learn from crowd labels directly [8,15]. The basic idea is employing generative models for the labeling processes of crowd labelers.…”

“…Our major contribution is a formalized framework for utilizing expert labels in crowdsourcing. Following a series of existing work [8,15,19], our work focuses on supervised classification problems.…”