A Structured Response to Misinformation

Abstract

The proliferation of misinformation in online news and its amplification by platforms are a growing concern, leading to numerous efforts to improve the detection of and response to misinformation. Given the variety of approaches, collective agreement on the indicators that signify credible content could allow for greater collaboration and data-sharing across initiatives. In this paper, we present an initial set of indicators for article credibility defined by a diverse coalition of experts. These indicators originate from both within an article's text as well as from external sources or article metadata. As a proof-of-concept, we present a dataset of 40 articles of varying credibility annotated with our indicators by 6 trained annotators using specialized platforms. We discuss future steps including expanding annotation, broadening the set of indicators, and considering their use by platforms and the public, towards the development of interoperable standards for content credibility.This paper is published under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. Authors reserve their rights to disseminate the work on their personal and corporate Web sites with the appropriate attribution.

Citation Needed: A Taxonomy and Algorithmic Assessment of Wikipedia's Verifiability

“…Crowdworkers and domain experts may disagree on classification tasks that require special knowledge [46]. However, Zhang et al [51] found that non-expert judgments about the characteristics of statements in news articles, such as whether a claim was well supported by the evidence provided, showed high inter-annotator agreement and high correlation with expert judgments. In the context of our study, this suggests that crowdworkers may find it relatively easier to provide reasons for citations than to decide which sentences require them in the first place.…”

SciLens: Evaluating the Quality of Scientific News Articles Using Social Media and Scientific Literature Indicators

“…A quotee can be an unnamed scientist or an unnamed study if the person or article being quoted is not disclosed (e.g., "researchers believe, " "most scientists think" and other so-called "weasel" words). Sources that are not specifically attributed such as these ones are as a general rule considered less credible than sources in which the quotee is named [62].…”

“…Recent work has demonstrated methods to automate the extraction of signals or indicators of article quality. These indicators are either expressed at a conceptual level [58] (e.g, balance of view points, respect of personal rights) or operationalized as features that can be computed from an article [62] (e.g., expert quotes or citations). Shu et al [55] describe an approach for detecting fake news on social media based on social and content indicators.…”

“…As a starting point, we adopt a large set of content-based quality indicators described by previous work. These indicators are: (i) title deceptiveness and sentiment: we consider if the title is "clickbait" that oversells the contents of an article in order to pique interest [41,60]; (ii) article readability: indicator of the level of education someone would need to easily read and understand the article [19]; and (iii) article length and presence of author byline [62].…”

Check-It: A plugin for Detecting and Reducing the Spread of Fake News and Misinformation on the Web

“…A few early studies tried to detect fake news based on linguistic features extracted from the text of news stories [20], [21], [27]. Recent studies have also shown that social networking features play a very important role in detecting fake news [24].…”