Risks and Rewards of Crowdsourcing Marketplaces

Chandler, Jesse; Paolacci, Gabriele; Mueller, Pam

doi:10.1007/978-1-4614-8806-4_30

Cited by 53 publications

(46 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several research studies have recommended using a HIT approval rate of 95% or greater (Berinsky, Huber, & Lenz, ; Chandler, Paolacci, & Mueller, ; Goodman et al, ; Paolacci, Chandler, & Ipeirotis, ; Peer et al, ). The HIT approval requirement of 1000 or more approved HITs was designed to have a meaningful base to determine the 95% HIT approval rate.…”

mentioning

confidence: 99%

Audit firm tenure, auditor familiarity, and trust: Effect on auditee whistleblowing reporting intentions

2018

View full text Add to dashboard Cite

Mandatory audit firm rotation has been researched for decades with resulting opposition as well as support. Research has mainly treated mandatory auditor rotation at the firm macro level. We submit the client relationship length is comprised of firm tenure and audit team continuity, or auditor familiarity. Increased tenure, at the interorganizational or firm level and interpersonal or individual level, has been shown to increase trust; and further, trust is positively related to employee voice, such as speaking up about fraud (whistleblowing). We conduct an experiment examining whether increased audit firm tenure and auditor familiarity leads to increased trust, which enhances the willingness to whistleblow. We find evidence that suggests auditor familiarity enhances trust, which, in turn, positively influences an employee's intentions to whistleblow. This has important implications for the profession and for future research exploring mandatory audit firm rotation; in particular, the need to include auditor familiarity as a construct.

show abstract

mentioning

confidence: 99%

Audit firm tenure, auditor familiarity, and trust: Effect on auditee whistleblowing reporting intentions

2018

View full text Add to dashboard Cite

show abstract

“…We distinguish the following sub-dimensions: -Description: This is the description of the work the requester asks the crowd to perform; it includes the instructions of how to perform the task and possible context information. The clarity and details of the description influence the way workers perform the task and hence the quality of its output [Chandler et al 2013]. -User interface: This is the software user interface workers use to perform the task.…”

Section: Crowdsourcing Quality Modelmentioning

confidence: 99%

Quality Control in Crowdsourcing

et al. 2018

View full text Add to dashboard Cite

Crowdsourcing enables one to leverage on the intelligence and wisdom of potentially large groups of individuals toward solving problems. Common problems approached with crowdsourcing are labeling images, translating or transcribing text, providing opinions or ideas, and similar -all tasks that computers are not good at or where they may even fail altogether. The introduction of humans into computations and/or everyday work, however, also poses critical, novel challenges in terms of quality control, as the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives and technological resources. This survey studies quality in the context of crowdsourcing along several dimensions, so as to define and characterize it and to understand the current state of the art. Specifically, this survey derives a quality model for crowdsourcing tasks, identifies the methods and techniques that can be used to assess the attributes of the model, and the actions and strategies that help prevent and mitigate quality problems. An analysis of how these features are supported by the state of the art further identifies open issues and informs an outlook on hot future research directions.

show abstract

“…Since the microtask paradigm in large scale crowdsourcing involves monotonous sequences of repetitive tasks, fatigue buildup can pose a potential problem to the quality of submitted work over time [9]. Furthermore, workers have been noted to be "satisficers" who, as they gain familiarity with the task and its acceptance thresholds, strive to do the minimal work possible to achieve these thresholds [8,51].…”

Section: Cscw '17mentioning

confidence: 99%

A Glimpse Far into the Future

Hata

Krishna

et al. 2017

Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing

View full text Add to dashboard Cite

Microtask crowdsourcing is increasingly critical to the creation of extremely large datasets. As a result, crowd workers spend weeks or months repeating the exact same tasks, making it necessary to understand their behavior over these long periods of time. We utilize three large, longitudinal datasets of nine million annotations collected from Amazon Mechanical Turk to examine claims that workers fatigue or satisfice over these long periods, producing lower quality work. We find that, contrary to these claims, workers are extremely stable in their quality over the entire period. To understand whether workers set their quality based on the task's requirements for acceptance, we then perform an experiment where we vary the required quality for a large crowdsourcing task. Workers did not adjust their quality based on the acceptance threshold: workers who were above the threshold continued working at their usual quality level, and workers below the threshold self-selected themselves out of the task. Capitalizing on this consistency, we demonstrate that it is possible to predict workers' long-term quality using just a glimpse of their quality on the first five tasks.

show abstract

Risks and Rewards of Crowdsourcing Marketplaces

Cited by 53 publications

References 58 publications

Audit firm tenure, auditor familiarity, and trust: Effect on auditee whistleblowing reporting intentions

Audit firm tenure, auditor familiarity, and trust: Effect on auditee whistleblowing reporting intentions

Quality Control in Crowdsourcing

A Glimpse Far into the Future

Contact Info

Product

Resources

About