Substance without citation: evaluating the online impact of grey literature

Abstract

Individuals and organisations producing information or knowledge for others sometimes need to be able to provide evidence of the value of their work in the same way that scientists may use journal Impact Factors and citations to indicate the value of their papers. There are many cases, however, when organisations are charged with producing reports but have no real way of measuring their impact, including when they are distributed free, do not attract academic citations and their sales cannot be tracked. Here, the Web Impact Report (WIRe) is proposed as a novel solution for this problem. A WIRe consists of a range of web-derived statistics about the frequency and geographic location of online mentions of an organisation's reports. WIRe data is typically derived from commercial search engines. This article defines the component parts of a WIRe and describes how to collect and analyse the necessary data. The process is illustrated with a comparison of the web impact of the reports of a large UK organisation. Although a formal evaluation was not conducted, the results suggest that WIRes can indicate different levels of web impact between reports and can reveal the type of online impact that the reports have.

Web indicators for research evaluation. Part 1: Citations and links to academic articles from the Web

“…The discussions below cover online presentations, syllabi and science blogs, although there is also some evidence that mentions in news websites and discussion forums may also be useful (Costas; Zahedi; Wouters, 2014; Thelwall, Haustein; Larivière; Sugimoto, 2013). Citations from online grey literature seem to be an additional useful source of evidence of the wider impact of research (Wilkinson;, but there do not seem to be any systematic studies of these.…”

Can the impact of grey literature be assessed? An investigation of UK government publications cited by articles and books

“…However, Google Scholar queries cannot be automated on a large scale, except for the facilities of Publish or Perish (Harzing, 2010) and it is therefore not suitable for large scale grey literature evaluations. Web queries have also been proposed for small sets of documents (Wilkinson, Sud, & Thelwall, 2014), but these do not necessarily reflect academic impact. Given the lack of an accepted solution for determining the academic impact of grey literature, this article proposes and demonstrates two new approaches.…”

Web indicators for research evaluation. Part 3: books and non standard outputs

“…Moreover, the main sources of citations to humanities books are other books (Thompson, 2002;Kousha;Thelwall, 2014). Even today, the Thomson Reuters Book Citation Index and Scopus index a relatively small number of books (60,000 1 and probably 70,000 2 as of July 2015, respectively) and this may cause problems for bibliometric analyses of books (e.g., Gorraiz; Purnell; Glänzel, 2013; Torres- Salinas et al, 2012Salinas et al, , 2013.…”