Ranking of Wikipedia articles in search engines revisited: Fair ranking for reasonable quality?

Abstract

This paper aims to review the fiercely discussed question of whether the ranking of Wikipedia articles in search engines is justified by the quality of the articles. After an overview of current research on information quality in Wikipedia, a summary of the extended discussion on the quality of encyclopedic entries in general is given. On this basis, a heuristic method for evaluating Wikipedia entries is developed and applied to Wikipedia articles that scored highly in a search engine retrieval effectiveness test and compared with the relevance judgment of jurors. The quality discussion regarding Wikipedia-not only in the mainstream media-is, as Prillinger's statement illustrates, very much focused on aspects of correctness in form and content (Giles, 2005). This narrowing of the quality debate does not meet the requirements of the complex discussion on lexicographical quality in general. In this discussion, correctness is just one facet in a set of criteria. In this article we shed new light on the question of lexicographic quality as our pragmatic research question is whether the overall quality of the retrieved articles, correctness included, justifies its ranking, or, in other words, is the ranking of the articles appropriate to their usefulness for the user of the search engine? Our research is also to be seen in the larger context of adding transparency to the question of how search engines deal with Wikipedia entries. Encyclopedias are generally recommended as a starting point for information research. Does the ranking in Wikipedia articles do this view justice?To provide some contextual background we summarize briefly the current research on information quality in Wikipedia as well as the extended discussion on the quality of encyclopedic entries in general. On this basis, we develop our own heuristic method for evaluating Wikipedia entries.

Evaluating the retrieval effectiveness of web search engines using a representative query sample

“…There are also studies which either compare search engines to other information systems (e.g., question answering systems (Vakkari, 2011) or use a combination of methods for measuring retrieval effectiveness and measurements of correlations between relevance judgments and other criteria, e.g., correlating relevance judgments for Wikipedia articles found in search engines with an assessment using lexical criteria (Lewandowski & Spree, 2011), or correlating relevance judgments with commercial intent of the documents found (Lewandowski, 2011b). There is also a trend towards carrying out more international and country-specific studies -see Lazarinis, Vilares, Tait, & Efthimiadis (2009) for an overview.…”

The People’s Encyclopedia Under the Gaze of the Sages: A Systematic Review of Scholarly Research on Wikipedia

“…Lewandowski and Spree (2011) found that Wikipedia results shown on search engines are quite dependent on the quality of articles; thus quality of articles has important direct consequences for its readership and existence on the web. Thus, Wikipedia has attracted significant attention from researchers to investigate this important aspect.…”

“…Google boost Wikipedia result mostly on first position but shows less Wikipedia links in total [sic]" (2009, p.1810). Lewandowski and Spree (2011) noted that Wikipedia results shown on search engines are quite dependent on the quality of articles.…”

Les jeunes, leurs enseignants et Wikipédia : représentations en tension autour d'un objet documentaire singulier

“…En effet, les internautes ont tendance à juger les pages de Wikipédia plus pertinentes que celles des autres sites figurant dans la liste des résultats [15]. C'est donc que l'encyclopédie bénéficie de représentations positives qui conduisent à sa consultation.…”