Crowd wisdom enhanced by costly signaling in a virtual rating system

Abstract

Costly signaling theory was developed in both economics and biology and has been used to explain a wide range of phenomena. However, the theory’s prediction that signal cost can enforce information quality in the design of new communication systems has never been put to an empirical test. Here we show that imposing time costs on reporting extreme scores can improve crowd wisdom in a previously cost-free rating system. We developed an online game where individuals interacted repeatedly with simulated services and rated them for satisfaction. We associated ratings with differential time costs by endowing the graphical user interface that solicited ratings from the users with “physics,” including an initial (default) slider position and friction. When ratings were not associated with differential cost (all scores from 0 to 100 could be given by an equally low-cost click on the screen), scores correlated only weakly with objective service quality. However, introducing differential time costs, proportional to the deviation from the mean score, improved correlations between subjective rating scores and objective service performance and lowered the sample size required for obtaining reliable, averaged crowd estimates. Boosting time costs for reporting extreme scores further facilitated the detection of top performances. Thus, human collective online behavior, which is typically cost-free, can be made more informative by applying costly signaling via the virtual physics of rating devices.

Experimenting With Online Governance

“…The implementation presented here is of in-browser WebGL via Unity. A specific implementation of this system for testing a virtual rating system was recently published (Tchernichovski et al, 2019). Beyond this, we acknowledge that, despite the wealth of existing tools, there is not yet a consensus regarding the best ways to connect such virtual environments to the experimental infrastructures we reviewed above.…”

“…However, it is often possible and practical to improve crowd wisdom in order to promote the early detection of trends (Tchernichovski et al, 2017). That is, crowd wisdom can be defined as the speed and accuracy of detecting a change (Tchernichovski et al, 2019). The early detection of trends can be particularly useful in online communities, where social feedback is continuous.…”

“…There are many concerns about the quality (and honesty) of such rating scores (Luca, 2016). A recent FSG experiment found that even without any conflicts of interest or incentives to cheat, pooled rating scores could explain only about 14% of the variance in the speed of ferries (Tchernichovski et al, 2019). However, pooled rating accuracy was about twice as high when ratings were submitted via a device that imposed time costs of a few seconds on reporting extreme scores.…”

“…Third, FSG experiments can be designed for testing the utility and risks of implementing machine learning for continuous optimization of rating system features. Such features may include, for example, the physics of a rating device (Tchernichovski et al, 2019), or the temporal resolution of a dashboard presenting trends (Tchernichovski et al, 2017), or both. About a decade ago, much of the software industry adopted continuous deployment, such that software is continually released and experimented with (Kevic et al, 2017).…”

Decision Neuroscience and Neuroeconomics: Recent Progress and Ongoing Challenges

“…Although these recent advances signal great promise for future neuroforecasting, we believe future work should go beyond traditional and conventional behavioral measures tied to choice, effort, or ratings (Lopez-Persem et al, 2017). For instance, ratings that incur a cost may provide a more honest metric of appeal in the aggregate (Tchernichovski et al, 2019); similarly, latent variables derived from computational models may provide more insight into the preferences of the aggregate (Clithero, 2018b(Clithero, , 2018a.…”