CanAmazon.com reviews help to assess the wider impacts of books?

Abstract

Although citation counts are often used to evaluate the research impact of academic publications, they are problematic for books that aim for educational or cultural impact. To fill this gap, this article assesses whether a number of simple metrics derived from Amazon.com reviews of academic books could give evidence about their impacts. Based upon a set of 2,739 academic monographs from 2008 and a set of 1,305 best-selling books in 15 Amazon.com academic subject categories, the existence of significant but low or moderate correlations between citations and numbers of reviews, combined with other evidence, suggests that online book reviews tend to reflect the wider popularity of a book rather than its academic impact, although there are substantial disciplinary differences. Metrics based upon online reviews are therefore recommended for the evaluation of books that aim at a wide audience inside or outside academia when it is important to capture the broader impacts of educational or cultural activities and when they cannot be manipulated in advance of the evaluation.

The Complexity of Measuring the Impact of Books

“…The diverse format types of books (such as monographs, edited volumes, major reference works and graduate level textbooks), where the analysis of their citation count alone is unlikely to fully reflect their value in research, teaching, and other professional activities; especially since books are not cited as often as articles [7]. [10] and Elsevier launched the Books Expansion Project; indexing books on its citation-indexing database called Scopus [3].…”

Libcitations, worldcat, cultural impact, and fame

“…As a result, citations and libcitations are not necessarily correlated. Zuccala and White () and Kousha and Thelwall () found correlations that were statistically significant but low. It is true that some books are both heavily cited and widely held, but other widely‐held books have citation records that can only be called unimpressive.…”

Reacting to Consecrating Science: What Might Amateurs Do?

“…These large data sets allow us to assess the impact of the new cosmologies on people who are actually engaging with the books but are not necessarily leaders of the movement. Indeed, researchers have begun to study popular book reviews, particularly on Amazon, recognizing that they can be rich sources of popular opinion (Reagle , 37–47; Allington , 258; Kousha and Thelwall ). For instance, Paul C. Gutjahr's research on Amazon reviews of the Left Behind series indicates that evangelical Christians may come to see popular novels as sacred texts, an indication that reviews can shed significant light on the popular consecration of ideas and texts (Gutjahr ).…”

“…The positive nature of the reviews may also arise from the fact that when texts are written for a popular audience, readers are likely already sympathetic to the text. Indeed, 80 percent of all Amazon book reviews are positive (Reagle , 65; Kousha and Thelwall , 569). Unless an author is highly controversial (e.g., Dawkins), potential readers who fundamentally disagree with the author are probably less likely to read and review the author's work.…”