Moderating with the Mob: Evaluating the Efficacy of Real-Time Crowdsourced Fact-Checking

Abstract


 
 
 Reducing the spread of false news remains a challenge for social media platforms, as the current strategy of using third-party fact- checkers lacks the capacity to address both the scale and speed of misinformation diffusion. Research on the “wisdom of the crowds” suggests one possible solution: aggregating the evaluations of ordinary users to assess the veracity of information. In this study, we investigate the effectiveness of a scalable model for real-time crowdsourced fact-checking. We select 135 popular news stories and have them evaluated by both ordinary individuals and professional fact-checkers within 72 hours of publication, producing 12,883 individual evaluations. Although we find that machine learning-based models using the crowd perform better at identifying false news than simple aggregation rules, our results suggest that neither approach is able to perform at the level of professional fact-checkers. Additionally, both methods perform best when using evaluations only from survey respondents with high political knowledge, suggesting reason for caution for crowdsourced models that rely on a representative sample of the population. Overall, our analyses reveal that while crowd-based systems provide some information on news quality, they are nonetheless limited—and have significant variation—in their ability to identify false news.
 
 

Research note: Fighting misinformation or fighting for information?

“…Few estimates are exempt of these limitations. Recently, Godel et al (2021), developed an algorithm selecting the most popular news from low-quality sources and had participants rating these news stories 72 hours after their publication at most. This overcomes the self-selection of news bias and reduces the delay between circulation and evaluation.…”

What is the Will of the People? Moderation Preferences for Misinformation

“…This delegates questions of facts to experts while reserving matters of subjective judgment to the more representative citizen panel. 3 The quality of lay raters' judgments of the holistic attribute of misinformation can be assessed in part by using the judgments of expert raters as a benchmark [3,6,27,51,52]. Previous studies indicate that panels of lay raters can perform reasonably well against this benchmark on published news articles, suggesting that this challenge may not be insurmountable for most articles [3,51].…”

“…The subset were selected based on a criterion that their headline or lede included a factual claim. The other 165 articles consisted of the most popular article each day from each of five categories: liberal mainstream news; conservative mainstream news; liberal low-quality news; conservative low-quality news; and low-quality news sites with no clear political orientation [27]. Five articles were selected on each of 33 days between November 13, 2019 and February 6, 2020.…”

“…The prevailing narrative is that the primary challenge in moderation decisions about misinformation is one of separating fact from fiction. Indeed, platforms have partnered with independent fact-checking organizations to verify facts [20] while considerable emphasis has been put on scaling up the fact-checking process to handle the large volume of content circulating on social media platforms [3,27,49,56]. The framing implies that platforms be judged by how "correct" their actions are.…”

"This is Fake News": Characterizing the Spontaneous Debunking from Twitter Users to COVID-19 False Information

“…Fact-checking by crowdsourcing exploits the wisdom of crowds. It is reported that their judgment has a high correlation with professional judgment in fact-checking (Epstein, Pennycook, and Rand 2020), although not as accurate as professionals yet (Godel et al 2021). While fack-checking by crowdsourcing is a promising complementary approach to trained fact-checkers, it still requires a great amount of time and resources if the crowd needs to verify a large amount of information.…”