A crowdsourcing quality control model for tasks distributed in parallel

Zhu, Shaojian; Kane, Shaun K.; Feng, Jinjuan; Sears, Andrew

doi:10.1145/2212776.2223826

Cited by 12 publications

(5 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A particular concern in the application of crowdsourcing, however, is to ensure the quality of work produced by crowd workers. Numerous methods have been investigated, including using a Gold Standard [36], machine learning classifiers [37], another crowd to assess the quality of the work [38], and associated data such as worker behaviour [39]. None of these approaches are entirely satisfactory in the context of software task estimation.…”

Section: Related Workmentioning

confidence: 99%

Playing Planning Poker in Crowds: Human Computation of Software Effort Estimates

Alhamed

Storer

2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

Reliable cost effective effort estimation remains a considerable challenge for software projects. Recent work has demonstrated that the popular Planning Poker practice can produce reliable estimates when undertaken within a software team of knowledgeable domain experts. However, the process depends on the availability of experts and can be time-consuming to perform, making it impractical for large scale or open source projects that may curate many thousands of outstanding tasks. This paper reports on a full study to investigate the feasibility of using crowd workers supplied with limited information about a task to provide comparably accurate estimates using Planning Poker. We describe the design of a Crowd Planning Poker (CPP) process implemented on Amazon Mechanical Turk and the results of a substantial set of trials, involving more than 5000 crowd workers and 39 diverse software tasks. Our results show that a carefully organised and selected crowd of workers can produce effort estimates that are of similar accuracy to those of a single expert.

show abstract

Section: Related Workmentioning

confidence: 99%

Playing Planning Poker in Crowds: Human Computation of Software Effort Estimates

Alhamed

Storer

2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

show abstract

“…Furthermore, in [19,21,22], the authors introduced the development and the strength of blockchain and described how to implement a safe and trustworthy crowdsourcing with privacy protection. Zhu, Kane et al [23] proposed a data-driven crowdsourcing quality control model for tasks distributed in parallel.…”

Section: Crowdsourcingmentioning

confidence: 99%

TFCrowd: a blockchain-based crowdsourcing framework with enhanced trustworthiness and fairness

Guo

2021

J Wireless Com Network

View full text Add to dashboard Cite

Blockchain technology has attracted considerable attention due to the boom of cryptocurrencies and decentralized applications. Among them, the emerging blockchain-based crowdsourcing is a typical paradigm, which gets rid of centralized cloud-servers and leverages smart contracts to realize task recommendation and reward distribution. However, there are still two critical issues yet to be solved urgently. First, malicious evaluation from crowdsourcing requesters will result in honest workers not getting the rewards they deserve even if they have provided valuable solutions. Second, unfair evaluation and reward distribution can lead to low enthusiasm for work. Therefore, the above problems will seriously hinder the development of blockchain-based crowdsourcing platforms. In this paper, we propose a new blockchain-based crowdsourcing framework with enhanced trustworthiness and fairness, named TFCrowd. The core idea of TFCrowd is utilizing a smart contract of blockchain as a trusted authority to fairly evaluate contributions and allocate rewards. To this end, we devise a reputation-based evaluation mechanism to punish the requester who behaves as “false-reporting” and a Shapley value-based method to distribute rewards fairly. By using our proposed schemes, TFCrowd can prevent malicious requesters from making unfair comments and reward honest workers according to their contributions. Extensive simulations and the experiment results demonstrate that TFCrowd can protect the interests of workers and distribute rewards fairly.

show abstract

“…Furthermore, in [17,19,20], the authors introduced the development and the strength of blockchain and described how to implement a safe and trustworthy crowdsourcing with privacy protection. Zhu, Kane et al [21] proposed a data-driven crowdsourcing quality control model for tasks distributed in parallel.…”

Section: Crowdsourcingmentioning

confidence: 99%

TFCrowd: A Blockchain-based Crowdsourcing Framework with Enhanced Trustworthiness and Fairness

Guo

2021

Preprint

View full text Add to dashboard Cite

Blockchain technology has attracted considerable attention due to the boom of cryptocurrencies and decentralized applications. Among them, the emerging blockchain-based crowdsourcing is a typical paradigm, which gets rid of centralized cloud-servers and leverages smart contracts to realize task recommendation and reward distribution. However, there are still two critical issues yet to be solved urgently. First, malicious evaluation from crowdsourcing requesters will result in honest workers not getting the rewards they deserve even providing valuable solutions. Second, unfair evaluation and reward distribution can lead to workers’ reluctance to work actively. Therefore, the above problems will seriously hinder the development of blockchain-based crowdsourcing platforms. In this paper, we propose a new blockchain-based crowdsourcing framework with enhanced trustworthiness and fairness, named TFCrowd. The core idea of TFCrowd is utilizing a smart contract of blockchain as a trusted authority to fairly evaluate contributions and allocate rewards. To this end, we devise a reputation-based evaluation mechanism to punish the requester who behaves as “false-reporting” and a Shapley value-based method to distribute rewards fairly. By using our proposed schemes, TFCrowd can prevent malicious requesters from making unfair comments and reward honest workers according to their contributions. Extensive simulations and the experiment results demonstrate that TFCrowd can protect the interests of workers and distribute rewards fairly.

show abstract

A crowdsourcing quality control model for tasks distributed in parallel

Cited by 12 publications

References 7 publications

Playing Planning Poker in Crowds: Human Computation of Software Effort Estimates

Playing Planning Poker in Crowds: Human Computation of Software Effort Estimates

TFCrowd: a blockchain-based crowdsourcing framework with enhanced trustworthiness and fairness

TFCrowd: A Blockchain-based Crowdsourcing Framework with Enhanced Trustworthiness and Fairness

Contact Info

Product

Resources

About