Worker Demographics and Earnings on Amazon Mechanical Turk

Hara, Kotaro; Adams, Abigail; Milland, Kristy; Savage, Saiph; Hanrahan, Benjamin V.; Bigham, Jeffrey P.; Callison-Burch, Chris

doi:10.1145/3290607.3312970

Cited by 41 publications

(29 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Turkers submitted 10 translations per task for $0.7 and $0.25 to rank 10 candidate translations, at an average rate to receive an equivalent full-time wage of $8.23/hour. This is markedly above the average wage for US workers of $3.01/hour discovered by Hara et al (2019). To ensure data quality and filter disfluencies or personal biases from Turkers, we then recruited bilingual postgraduate students, native speakers of the task language, to judge if the best chosen translation from Turk was satisfactory or required rewriting.…”

Section: Data Collectionmentioning

confidence: 99%

Bootstrapping a Crosslingual Semantic Parser

Sherborne

Lapata

2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Recent progress in semantic parsing scarcely considers languages other than English but professional translation can be prohibitively expensive. We adapt a semantic parser trained on a single language, such as English, to new languages and multiple domains with minimal annotation. We query if machine translation is an adequate substitute for training data, and extend this to investigate bootstrapping using joint training with English, paraphrasing, and multilingual pre-trained models. We develop a Transformer-based parser combining paraphrases by ensembling attention over multiple encoders and present new versions of ATIS and Overnight in German and Chinese for evaluation. Experimental results indicate that MT can approximate training data in a new language for accurate parsing when augmented with paraphrasing through multiple MT engines. Considering when MT is inadequate, we also find that using our approach achieves parsing accuracy within 2% of complete translation using only 50% of training data. 1

show abstract

Section: Data Collectionmentioning

confidence: 99%

Bootstrapping a Crosslingual Semantic Parser

Sherborne

Lapata

2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

show abstract

“…The projected hourly rate is: (0.55 USD/HIT * 3600 sec/hr ) / (235.322 sec/HIT) = 8.41 USD/hr. As such, our average rate of pay is well above the standard average rate for similar crowdsourcing tasks [46,87]. Of course, some workers will have taken significantly longer to complete this HIT, making their rate of pay noticeably lower.…”

Section: Crowdwork Overviewmentioning

confidence: 76%

“…An aspect of our findings that we did not anticipate centers on the demographics of workers on AMT. While extensive literature discusses various social and economic demographic characteristics of Turkers [31,46], our work is the first study (to our knowledge) to discuss the familiarity of the AMT crowd with the interests/needs of people who are blind. We were surprised to learn that many Turkers expressed intimate familiarity with this population (via marriage, profession, family, personal experience, etc.).…”

Section: Engaging Crowdworkers In Accessibility-related Tasksmentioning

confidence: 99%

"I Hope This Is Helpful"

Simons

Gurari

Fleischmann

2020

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

AI image captioning challenges encourage broad participation in designing algorithms that automatically create captions for a variety of images and users. To create large datasets necessary for these challenges, researchers typically employ a shared crowdsourcing task design for image captioning. This paper discusses findings from our thematic analysis of 1,064 comments left by Amazon Mechanical Turk workers using this task design to create captions for images taken by people who are blind. Workers discussed difficulties in understanding how to complete this task, provided suggestions of how to improve the task, gave explanations or clarifications about their work, and described why they found this particular task rewarding or interesting. Our analysis provides insights both into this particular genre of task as well as broader considerations for how to employ crowdsourcing to generate large datasets for developing AI algorithms.

show abstract

“…Large-scale datasets [8,37] show that cultivating good workers produces high quality data at low cost. [22]. For "image transcription", the median wage is $1.13/hr over 150K tasks.…”

Section: Annotation Cost and Worker Feedbackmentioning

confidence: 99%

Block Annotation: Better Image Annotation With Sub-Image Decomposition

Lin

Upchurch

Bala

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Image datasets with high-quality pixel-level annotations are valuable for semantic segmentation: labelling every pixel in an image ensures that rare classes and small objects are annotated. However, full-image annotations are expensive, with experts spending up to 90 minutes per image. We propose block sub-image annotation as a replacement for full-image annotation. Despite the attention cost of frequent task switching, we find that block annotations can be crowdsourced at higher quality compared to full-image annotation with equal monetary cost using existing annotation tools developed for full-image annotation. Surprisingly, we find that 50% pixels annotated with blocks allows semantic segmentation to achieve equivalent performance to 100% pixels annotated. Furthermore, as little as 12% of pixels annotated allows performance as high as 98% of the performance with dense annotation. In weakly-supervised settings, block annotation outperforms existing methods by 3-4% (absolute) given equivalent annotation time. To recover the necessary global structure for applications such as characterizing spatial context and affordance relationships, we propose an effective method to inpaint block-annotated images with high-quality labels without additional human effort. As such, fewer annotations can also be used for these applications compared to full-image annotation.

show abstract

Worker Demographics and Earnings on Amazon Mechanical Turk

Cited by 41 publications

References 5 publications

Bootstrapping a Crosslingual Semantic Parser

Bootstrapping a Crosslingual Semantic Parser

"I Hope This Is Helpful"

Block Annotation: Better Image Annotation With Sub-Image Decomposition

Contact Info

Product

Resources

About