Just in Time

Abstract

Many modern data analytics applications in areas such as crisis management, stock trading, and healthcare, rely on components capable of nearly real-time processing of streaming data produced at varying rates. In addition to automatic processing methods, many tasks involved in those applications require further human assessment and analysis. However, current crowdsourcing platforms and systems do not support stream processing with variable loads. In this paper, we investigate how incentive mechanisms in competition based crowdsourcing can be employed in such scenarios. More specifically, we explore techniques for stimulating workers to dynamically adapt to both anticipated and sudden changes in data volume and processing demand, and we analyze effects such as data processing throughput, peakto-average ratios, and saturation effects. To this end, we study a wide range of incentive schemes and utility functions inspired by real world applications. Our large-scale experimental evaluation with more than 900 participants and more than 6200 hours of work spent by crowd workers demonstrates that our competition based mechanisms are capable of adjusting the throughput of online workers and lead to substantial on-demand performance boosts.

Is Virtual Citizen Science A Game?

“…While the game-based project did introduce cheating as a strategy among players, overall (excluding the cheating behaviour) there was no significant difference in performance [Prestopnik et al 2014]. Similar findings have been made in other domains, around adding competitive elements in crowdsourcing to improve volunteer output without harming the quality of the data they produce [Rokicki et al 2016]. Taking a somewhat different approach, the effects of collaborative and competitive game elements have been studied, based on how they affect player engagement and accuracy in games with a purpose (GWAPs).…”