No Workflow Can Ever Be Enough

Abstract

The dominant crowdsourcing infrastructure today is the workflow, which decomposes goals into small independent tasks. However, complex goals such as design and engineering have remained stubbornly difficult to achieve with crowdsourcing workflows. Is this due to a lack of imagination, or a more fundamental limit? This paper explores this question through in-depth case studies of 22 workers across six workflow-based crowd teams, each pursuing a complex and interdependent web development goal. We used an inductive mixed method approach to analyze behavior trace data, chat logs, survey responses and work artifacts to understand how workers enacted and adapted the crowdsourcing workflows. Our results indicate that workflows served as useful coordination artifacts, but in many cases critically inhibited crowd workers from pursuing realtime adaptations to their work plans. However, the CSCW and organizational behavior literature argues that all sufficiently complex goals require open-ended adaptation. If complex work requires adaptation but traditional static crowdsourcing workflows can't support it, our results suggest that complex work may remain a fundamental limitation of workflow-based crowdsourcing infrastructures.

The Life Course: An interdisciplinary framework for broadening the scope of research on crowdwork

“…• Quantitatively driven, sequential QUAN-> qual studies, where the major data collection effort focused on a questionnaire survey, followed by a small number of interviews (Rani & Furrer, 2020). • Quantitatively driven, concurrent qual+QUAN designs draw on the analysis of digital behaviour trace and log data as well as work artefacts collected from the platform, coupled with a questionnaire survey of crowdworkers (Retelny et al, 2017). • Qualitatively driven, sequential QUAL-> QUAL -> quan design studies which start off with observations of crowdworkers completing tasks, followed by interviews, analysis of workers' performance on the tasks, and a questionnaire survey (Vashistha et al, 2018).…”

“…Only three of 14 papers studied online freelancing platforms, in particular Upwork and Freelancer (Al-Ani & Stumpp, 2016;Feldman et al, 2017;Graham et al, 2017;Retelny et al, 2017).…”

“…US and/or India (Gadiraju et al, 2017;Gray et al, 2016;Khanna et al, 2010;Kingsley et al, 2015); or blind, lowincome crowdworkers (Vashistha et al, 2018). In most mixed-method studies, the sample size ranged from 100 to 502 participants; only one study drew on a significantly larger sample of 6338 participants (Gray et al, 2016), whilst three of the 16 mixed-method studies used significantly smaller samples of 22, 32 and 39 participants (Gerber & Krzywdzinski, 2019;Retelny et al, 2017;Vashistha et al, 2018).…”

Exploring Self-organisation in Crowd Teams

“…This approach is indeed appropriate for microtask crowdsourcing, where the crowdsourcing task can be clearly delimited to discrete parts and given to specific workers with specific roles. However, when it comes to more complex work, which is usually the type of work that crowd teams are called to address, such algorithms can stifle creativity and initiative-taking, as indicated by recent research in management sciences [8] and crowdsourcing [6,18].…”

“…Similarly to most existing algorithm-based methods for crowd management (see for example [11,13]), our previous method suffered from one important disadvantage: not actively involving the workers in the process, but rather assigning them directly to a task or to a team. However, as latest research in management sciences [8] and also crowdsourcing [18] indicates, too close a monitoring can stifle worker creativity and initiative-taking: two features that are absolutely necessary in creative, complex teamwork.…”