Crowdsourcing in Computer Vision

Kovashka, Adriana; Russakovsky, Olga; Li, Feifei; Grauman, Kristen

doi:10.1561/0600000071

Cited by 74 publications

(47 citation statements)

References 110 publications

(162 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This paper complements and extends the existing understanding of crowd work earnings using a data-driven approach. Our research focuses on Amazon Mechanical Turk (AMT), one of the largest micro-crowdsourcing markets, that is widely used by industry [34,48] and the HCI community, as well as by other research areas such as NLP and computer vision [15,45]. At the core of our research is an unprecedented amount of worker log data collected by the Crowd Workers Chrome plugin [14] between Sept 2014 to Jan 2017.…”

Section: Introductionmentioning

confidence: 99%

A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk

Hara

Adams

Milland

et al. 2018

Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

379

266

View full text Add to dashboard Cite

A growing number of people are working as part of on-line crowd work. Crowd work is often thought to be low wage work. However, we know little about the wage distribution in practice and what causes low/high earnings in this setting. We recorded 2,676 workers performing 3.8 million tasks on Amazon Mechanical Turk. Our task-level analysis revealed that workers earned a median hourly wage of only ~$2/h, and only 4% earned more than $7.25/h. While the average requester pays more than $11/h, lower-paying requesters post much more work. Our wage calculations are influenced by how unpaid work is accounted for, e.g., time spent searching for tasks, working on tasks that are rejected, and working on tasks that are ultimately not submitted. We further explore the characteristics of tasks and working patterns that yield higher hourly wages. Our analysis informs platform design and worker tools to create a more positive future for crowd work. Figure 12. (a) Hourly wage distributions of seven HIT categories provided by Gadiraju et al. [25] (with an additional category Research). (b) Strip plots showing median hourly wages of HITs associated with the topical keywords in Table 4.

show abstract

Section: Introductionmentioning

confidence: 99%

A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk

Hara

Adams

Milland

et al. 2018

Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

379

266

View full text Add to dashboard Cite

show abstract

“…• Crowd-sourcing evaluation (Kovashka et al, 2016): categorizing user behaviors during crowd-sourced modeling and vandalism detection process (Neis et al, 2012).…”

Section: Positioning and Contributionsmentioning

confidence: 99%

A Learning Approach to Evaluate the Quality of 3D City Models

Ennafii¹,

Bris²,

Lafarge³

et al. 2019

photogramm eng remote sensing

View full text Add to dashboard Cite

The automatic generation of three-dimensional (3D) building models from geospatial data is now a standard procedure. An abundance of literature covers the last two decades, and several solutions are now available. However, urban areas are very complex environments. Inevitably, practitioners still have to visually assess, at a city-scale, the correctness of these models and detect frequent reconstruction errors. Such a process relies on experts and is highly time-consuming, with approximately two hours/km 2 per expert. This work proposes an approach for automatically evaluating the quality of 3D building models. Potential errors are compiled in a novel hierarchical and versatile taxonomy. This allows, for the first time, to disentangle fidelity and modeling errors, whatever the level of details of the modeled buildings. The quality of models is predicted using the geometric properties of buildings and, when available, Very High Resolution images and Digital Surface Models. A baseline of handcrafted, yet generic, features is fed into a Random Forest classifier. Both multiclass and multilabel cases are considered: due to the interdependence between classes of errors, it is possible to retrieve all errors at the same time while simply predicting correct and erroneous buildings. The proposed framework was tested on three distinct urban areas in France with more than 3000 buildings. 80%–99% F-score values are attained for the most frequent errors. For scalability purposes, the impact of the urban area composition on the error prediction was also studied, in terms of transferability, generalization, and representativeness of the classifiers. It showed the necessity of multimodal remote sensing data and mixing training samples from various cities to ensure a stability of the detection ratios, even with very limited training set sizes.

show abstract

“…Kovashka et al [15] provide a thorough review of crowdsourcing techniques for computer vision. The Dawid-Skene (DS) model [5] is the standard probabilistic model for multiclass label inference from multiple annotations.…”

Section: Related Workmentioning

confidence: 99%

Lean Multiclass Crowdsourcing

Horn¹,

Branson

Loarie³

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

View full text Add to dashboard Cite

We introduce a method for efficiently crowdsourcing multiclass annotations in challenging, real world image datasets. Our method is designed to minimize the number of human annotations that are necessary to achieve a desired level of confidence on class labels. It is based on combining models of worker behavior with computer vision. Our method is general: it can handle a large number of classes, worker labels that come from a taxonomy rather than a flat list, and can model the dependence of labels when workers can see a history of previous annotations. Our method may be used as a drop-in replacement for the majority vote algorithms used in online crowdsourcing services that aggregate multiple human annotations into a final consolidated label. In experiments conducted on two real-life applications we find that our method can reduce the number of required annotations by as much as a factor of 5.4 and can reduce the residual annotation error by up to 90% when compared with majority voting. Furthermore, the online risk estimates of the models may be used to sort the annotated collection and minimize subsequent expert review effort.

show abstract

Crowdsourcing in Computer Vision

Cited by 74 publications

References 110 publications

A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk

A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk

A Learning Approach to Evaluate the Quality of 3D City Models

Lean Multiclass Crowdsourcing

Contact Info

Product

Resources

About