Measuring the quality of scientific references in Wikipedia: an analysis of more than 115M citations to over 800 000 scientific articles

Abstract

Wikipedia is a widely used online reference work which cites hundreds of thousands of scientific articles across its entries. The quality of these citations has not been previously measured, and such measurements have a bearing on the reliability and quality of the scientific portions of this reference work. Using a novel technique, a massive database of qualitatively described citations, and machine learning algorithms, we analyzed 1,923,575 Wikipedia articles which cited a total of 824,298 scientific articles in our database, and found that most scientific articles cited by Wikipedia articles are uncited or untested by subsequent studies, and the remainder show a wide variability in contradicting or supporting evidence. Additionally, we analyzed 51,804,643 scientific articles from journals indexed in the Web of Science and found that similarly most were uncited or untested by subsequent studies, while

scite: a smart citation index that displays the context of citations and classifies their intent using deep learning

“…They found that negative citations are rare, even among works that have been retracted. In another example, Nicholson et al ( 46 ) examined scientific papers cited in Wikipedia articles and found that – like the scientific literature as a whole – the vast majority presented findings that have not been subsequently verified. Similar analyses could also be applied to articles in the popular press.…”

Scite

“…By way of example, the article, "PRESS peer review of electronic search strategies: 2015 guideline statement" [23], has been cited by 493 articles in PubMed, 835 publications with 590 extracted citation statements in Scite, 740 publications in Scopus, 671 publications in Web of Science, and 1,078, according to Google Scholar (as of June 4, 2021). As Scite's founders note, "Our results should be considered with caution given the limitations of the model precision, the current limited coverage of articles analyzed by scite, and the fact that articles that could not be linked to a DOI in the data set were excluded" [14].…”

“…The use and citation of disputed articles can impact research trends, clinical practice, and patient health and well-being [ 13 , 7 ]. It is important to note here that, because “(a) scite classifies citation statements at the level of the claim, not the full paper, and (b) the citing article making the contradicting claim itself could be without merit,” “a contradicting citation statement does not necessarily mean the cited paper is wrong” [ 14 ].…”

“…The Scite corpus has been used to check the quality of references cited in Wikipedia articles [ 14 ]. Scite data could be used in reviews looking at how datasets, software, and research tools are used by restricting a citation search to publications citing tools in their methods sections [ 20 ].…”

“…As Scite's founders note, “Our results should be considered with caution given the limitations of the model precision, the current limited coverage of articles analyzed by scite, and the fact that articles that could not be linked to a DOI in the data set were excluded” [ 14 ].…”