Studying the “Wisdom of Crowds” at Scale

Abstract

In a variety of problem domains, it has been observed that the aggregate opinions of groups are often more accurate than those of the constituent individuals, a phenomenon that has been dubbed the “wisdom of the crowd”. However, due to the varying contexts, sample sizes, methodologies, and scope of previous studies, it has been difficult to gauge the extent to which conclusions generalize. To investigate this question, we carried out a large online experiment to systematically evaluate crowd performance on 1,000 questions across 50 topical domains. We further tested the effect of different types of social influence on crowd performance. For example, in one condition, participants could see the cumulative crowd answer before providing their own. In total, we collected more than 500,000 responses from nearly 2,000 participants. We have three main results. First, averaged across all questions, we find that the crowd indeed performs better than the average individual in the crowd—but we also find substantial heterogeneity in performance across questions. Second, we find that crowd performance is generally more consistent than that of individuals; as a result, the crowd does considerably better than individuals when performance is computed on a full set of questions within a domain. Finally, we find that social influence can, in some instances, lead to herding, decreasing crowd performance. Our findings illustrate some of the subtleties of the wisdom-of-crowds phenomenon, and provide insights for the design of social recommendation platforms.

Blending NLP and Machine Learning for the Development of Winograd Schemas

“…Here, WinoFusion is taking the bonus factor to another level by showing the Qualificators the current amounts given to the crowd through a bonus banner. As other research showed [31], these incorporated gamification aspects can encourage and motivate workers to keep themselves busy.…”

“…Through better evaluation controls, WinoFusion adds to the WinoFlexi mechanisms to enhance quality and improve worker's experience. Via advanced collaboration techniques, it takes advantage of the variability that generally stems from Winventor's schemas and the "wisdom of the crowd" [31], which is arguably better than any individual worker's opinion.…”

“…Compared to WinoFlexi, within WinoFusion, every examined schema is evaluated by at least three Qualificators through an automatic procedure that takes place on specific days of the Weekend, set by the super administrator -more on the administrator's role can be found in the original WinoFlexi paper. Nevertheless, this evaluation mechanism adds to the quality of the developed schemas because of the "wisdom-of-the-crowd" factor [31]. After that, each schema is evaluated by our Supervisors and accordingly flagged as valid-finished or valid-pending.…”

“…Comments Banner: The ultimate goal is for WinoFusion to act as a collaborative task wherein multiple workers work together. The futility of this idea is to use the crowd's wisdom factor as it performs better than any average individual [31]. Participants who receive information about other people's choices behave better, leading to better results.…”

With a little help from my friends? The impact of social networks on citizens' forecasting ability

“…In a nutshell, the general idea behind this thesis can be stated as follows: ‘if error is randomly distributed, then in the aggregation of multitudinous judgments, error cancels out and truth remains’ (Landemore, 2012, p. 291; see also Page & Shapiro, 1992). The miracle of aggregation theorem is an example of the ‘wisdom of crowds’ (WOC) principle according to which groups take more enlightened decisions and show better judgment than any of their individual members taken separately (Simoiu et al., 2019; Surowiecki, 2004). This has been demonstrated repeatedly in the case of voters' expectations regarding election outcomes (see, e.g., Miller et al., 2012; Murr, 2017; Temporão et al., 2019).…”