Citation Needed: A Taxonomy and Algorithmic Assessment of Wikipedia's Verifiability

Abstract

Wikipedia is playing an increasingly central role on the web, and the policies its contributors follow when sourcing and fact-checking content affect million of readers. Among these core guiding principles, verifiability policies have a particularly important role. Verifiability requires that information included in a Wikipedia article be corroborated against reliable secondary sources. Because of the manual labor needed to curate and fact-check Wikipedia at scale, however, its contents do not always evenly comply with these policies. Citations (i.e. reference to external sources) may not conform to verifiability requirements or may be missing altogether, potentially weakening the reliability of specific topic areas of the free encyclopedia. In this paper, we aim to provide an empirical characterization of the reasons why and how Wikipedia cites external sources to comply with its own verifiability guidelines. First, we construct a taxonomy of reasons why inline citations are required by collecting labeled data from editors of multiple Wikipedia language editions. We then collect a large-scale crowdsourced dataset of Wikipedia sentences annotated with categories derived from this taxonomy. Finally, we design and evaluate algorithmic models to determine if a statement requires a citation, and to predict the citation reason based on our taxonomy. We evaluate the robustness of such models across different classes of Wikipedia articles of varying quality, as well as on an additional dataset of claims annotated for fact-checking purposes.

Quantifying Engagement with Citations on Wikipedia

“…Automated or semi-automated tools [17,39,44] can help improve user experience [29,69], content variety [42,67], and quality [1,20,28]. The reliability of Wikipedia can also be improved automatically, e.g., by finding potential citations [15] and Wikipedia statements in need of evidence [48]. The insights from our work can help improve Wikipedia via new citations with which users would be more likely to interact.…”

Claim Check-Worthiness Detection as Positive Unlabelled Learning

“…These works are focused solely on the political domain, using data from political TV shows, speeches, and debates. In contrast, in this work we study the claim checkworthiness detection problem across three domains which have publicly available data: Twitter (Zubiaga et al, 2017), political speeches (Atanasova et al, 2018), and Wikipedia (Redi et al, 2019).…”

“…Citation Needed Detection Wikipedia citation needed detection has been investigated recently in (Redi et al, 2019). The authors present a dataset of sentences from Wikipedia labelled for whether or not they have a citation attached to them.…”

“…Wikipedia Citations We use the dataset from Redi et al (2019) for citation needed detection. (Redi et al, 2019).…”

“…There are multiple isolated lines of research which have studied variations of this problem. Figure 1 provides examples from three tasks which are studied in this work: rumour detection on Twitter (Zubiaga et al, 2016, check-worthiness ranking in political debates and speeches (Atanasova et al, 2018;Elsayed et al, 2019;Barrón-Cedeño et al, 2020), and citation needed detection on Wikipedia (Redi et al, 2019). Each task is concerned with a shared underlying problem: detecting claims which war-rant further verification.…”