A review and experimental analysis of active learning over crowdsourced data

Abstract

Training data creation is increasingly a key bottleneck for developing machine learning, especially for deep learning systems. Active learning provides a cost-effective means for creating training data by selecting the most informative instances for labeling. Labels in real applications are often collected from crowdsourcing, which engages online crowds for data labeling at scale. Despite the importance of using crowdsourced data in the active learning process, an analysis of how the existing active learning approaches behave over crowdsourced data is currently missing. This paper aims to fill this gap by reviewing the existing active learning approaches and then testing a set of benchmarking ones on crowdsourced datasets. We provide a comprehensive and systematic survey of the recent research on active learning in the hybrid human–machine classification setting, where crowd workers contribute labels (often noisy) to either directly classify data instances or to train machine learning models. We identify three categories of state of the art active learning methods according to whether and how predefined queries employed for data sampling, namely fixed-strategy approaches, dynamic-strategy approaches, and strategy-free approaches. We then conduct an empirical study on their cost-effectiveness, showing that the performance of the existing active learning approaches is affected by many factors in hybrid classification contexts, such as the noise level of data, label fusion technique used, and the specific characteristics of the task. Finally, we discuss challenges and identify potential directions to design active learning strategies for hybrid classification problems.

Assessing the Image Concept Drift at the OBSEA Coastal Underwater Cabled Observatory

“…This process can be further improved by combining active learning with the crowd sourcing data and labelling, produced by citizen science activities as discussed in Støttrup et al (2018) andDiBattista et al (2021). In this case, several categories spanning from students to professional fisherman or amatorial divers can efficiently contribute to the labelling of the acquired images and combine this effort with active learning techniques to select the most relevant images for training/updating the classifier, as discussed in Sayin et al (2021).…”

Semantic Segmentation of Terrestrial Laser Scans of Railway Catenary Arches: A Use Case Perspective

“…Such an approach could be feasible as components of the catenary arch have strong geometric shapes. An alternative approach is to use an active learning paradigm [ 50 ] for reducing the labelling cost. It is possible to leverage the trained model in this feat for the human-in-loop approach.…”

Synthesis optimization and adsorption modeling of biochar for pollutant removal via machine learning

“…In addition, modern machine learning models are extremely complex, and it is appropriate to optimize models by iteration. Active learning tries to solve the labeling bottleneck of samples, and actively selects the most valuable unlabeled samples for labeling first, so as to achieve excellent model accuracy with as few labeled samples as possible (Faulon and Faure 2021;Sayin et al 2021). Active learning could overcome the shortcomings of traditional machine learning to some extent.…”