Playing Planning Poker in Crowds: Human Computation of Software Effort Estimates

Abstract

Reliable cost effective effort estimation remains a considerable challenge for software projects. Recent work has demonstrated that the popular Planning Poker practice can produce reliable estimates when undertaken within a software team of knowledgeable domain experts. However, the process depends on the availability of experts and can be time-consuming to perform, making it impractical for large scale or open source projects that may curate many thousands of outstanding tasks. This paper reports on a full study to investigate the feasibility of using crowd workers supplied with limited information about a task to provide comparably accurate estimates using Planning Poker. We describe the design of a Crowd Planning Poker (CPP) process implemented on Amazon Mechanical Turk and the results of a substantial set of trials, involving more than 5000 crowd workers and 39 diverse software tasks. Our results show that a carefully organised and selected crowd of workers can produce effort estimates that are of similar accuracy to those of a single expert.

Agile Effort Estimation: Comparing the Accuracy and Efficiency of Planning Poker, Bucket System, and Affinity Estimation Methods

“…In other words, Planning Poker is proven to be a more time-consuming method than both Bucket System and Affinity Estimation. Assuming that Planning Poker is a consensus-based technique based on intensive communication and is known for its time-consuming potential [31,5,67,68], the result is somewhat expected. From this point of view, our results are in line with the papers in the literature that argue that Planning Poker can be time-consuming to perform [66,68].…”

“…Gandomani et al [31] state that reaching group consensus on the user story size in Planning Poker can be difficult and time-consuming in practice. In their work, Alhamed and Storer [5] investigated the use of Planning Poker in crowdsourcing. They found that Planning Poker can be time-consuming and impractical for large projects requiring assessing thousands of outstanding tasks.…”

“…What is more, Altaleb et al [8] found that Expert Judgment and Planning Poker remain one of the most popular estimation techniques in mobile app development as well. In their paper, Alhamed and Storer [5] state that Planning Poker can produce reliable estimates with knowledgeable domain experts. The challenge of Planning Poker is that it can be time-consuming in practice and can have difficulties with reaching an agreement in time [31,81].…”

“…Planning Poker has been used in the software industry and teaching environment [40,31,19,44]. It can provide reliable estimates compared to single expert estimation [5]. Labedzki et al [40] presented a study based on three different reference agile projects with a mature team, different team sizes, project lengths and user stories that differed in size and how they were defined.…”

Prominence convergence in the strategy coordination of crowdsourcing workers

“…In real word, the final results of the tasks may have strong relation to the complex social relationships of workers because workers may coordinate with each other and adjust their strategies in the execution process. For example, in [6], Alhamed et al designed a process that applies the wisdom of crowd to software task efforts estimation method, i.e., every crowd worker would interact with other workers to evaluate task efforts and conduct a new round of evaluation on this basis. After multiple rounds of evaluation, the final result demonstrates that crowd workers can produce estimates of comparable accuracy to those of experts.…”