Scalable and efficient learning from crowds with Gaussian processes

Abstract

Over the last few years, multiply-annotated data has become a very popular source of information. Online platforms such as Amazon Mechanical Turk have revolutionized the labelling process needed for any classification task, sharing the effort between a number of annotators (instead of the classical single expert). This crowdsourcing approach has introduced new challenging problems, such as handling disagreements on the annotated samples or combining the unknown expertise of the annotators. Probabilistic methods, such as Gaussian Processes (GP), have proven successful to model this new crowdsourcing scenario. However, GPs do not scale up well with the training set size, which makes them prohibitive for medium-to-large datasets (beyond 10K training instances). This constitutes a serious limitation for current real-world applications. In this work, we introduce two scalable and efficient GP-based crowdsourcing methods that allow for processing previously-prohibitive datasets. The first one is an efficient and fast approximation to GP with squared exponential (SE) kernel. The second allows for learning a more flexible kernel at the expense of a heavier training (but still scalable to large datasets). Since the latter is not a GP-SE approximation, it can be also considered as a whole new scalable and efficient crowdsourcing method, useful for any dataset size. Both methods use Fourier features and variational inference, can predict the class of new samples, and estimate the expertise of the involved annotators. A complete experimentation compares them with state-of-the-art probabilistic approaches in synthetic and real crowdsourcing datasets of different sizes. They stand out as the best performing approach for large scale problems. Moreover, the second method is competitive with the current state-of-the-art for small datasets.

Scalable Variational Gaussian Processes for Crowdsourcing: Glitch Detection in LIGO

“…. There exist other sparse GP approximations which alternatively rely on Fourier features[30], and which have been already used for crowdsourcing problems[31]. In the experiments, the proposed method will be shown to clearly outperform these alternative approaches too.…”

“…However, the poor scalability of GPs hampered the wide adoption of these approaches in practice. This motivated the development of the so-called RFF and VFF algorithms, which leverage Random Fourier features approximations to GPs to propose two more scalable GPbased crowdsourcing methods [31]. These approaches significantly improve the scalability, reducing it from cubic O(N 3 ) to linear O(N D 2 ) (with D the number of Fourier frequencies used, D N , see [31]).…”

“…This motivated the development of the so-called RFF and VFF algorithms, which leverage Random Fourier features approximations to GPs to propose two more scalable GPbased crowdsourcing methods [31]. These approaches significantly improve the scalability, reducing it from cubic O(N 3 ) to linear O(N D 2 ) (with D the number of Fourier frequencies used, D N , see [31]). In practice, this implies moving from manageable data sets of N ≈ 10 4 up to N ≈ 10 5 /10 6 .…”

“…Moreover, figure 11 shows that their test performance is pretty far from that of the novel SVGPCR. Indeed, the logistic regression model underlying Raykar is not sufficient for the nonlinear problem at hand, and the GP approximation provided by RFF is known to be poor when the dimensionality of the problem is high [31] (like here, where we are working with 256 features, recall section 3). The rest of methods are also clearly outperformed, since their 0 2 5 0 0 0 5 0 0 0 0 7 5 0 0 0 1 0 0 0 0 0 1 2 5 0 0 0 1 5 0 0 0 0 The mean over five independent runs is shown.…”

Learning from crowds in digital pathology using scalable variational Gaussian processes

“…An analogous crowdsourcing framework has been also used with more expressive classifiers such as deep neural networks 9 , 10 . Gaussian processes were also introduced for crowdsourcing with sound results across different domains 11 – 13 . These models are Bayesian and non-parametric, making them suitable to learn good models without the need for very large labeled datasets.…”