Overcoming the J-shaped distribution of product reviews

Abstract

Introduction While product review systems that collect and disseminate opinions about products from recent buyers (Table 1) are valuable forms of word-of-mouth communication, evidence suggests that they are overwhelmingly positive. Kadet notes that most products receive almost five stars. Chevalier and Mayzlin also show that book reviews on Amazon and Barnes & Noble are overwhelmingly positive. Is this because all products are simply outstanding? However, a graphical representation of product reviews reveals a J-shaped distribution (Figure 1) with mostly 5-star ratings, some 1-star ratings, and hardly any ratings in between. What explains this J-shaped distribution? If products are indeed outstanding, why do we also see many 1-star ratings? Why aren't there any product ratings in between? Is it because there are no "average" products? Or, is it because there are biases in product review systems? If so, how can we overcome them? The J-shaped distribution also creates some fundamental statistical problems. Conventional wisdom assumes that the average of the product ratings is a sufficient proxy of product quality and product sales. Many studies used the average of product ratings to predict sales. However, these studies showed inconsistent results: some found product reviews to influence product sales, while others did not. The average is statistically meaningful only when it is based on a unimodal distribution, or when it is based on a symmetric bimodal distribution. However, since product review systems have an asymmetric bimodal (J-shaped) distribution, the average is a poor proxy of product quality. This report aims to first demonstrate the existence of a J-shaped distribution, second to identify the sources of bias that cause the J-shaped distribution, third to propose ways to overcome these biases, and finally to show that overcoming these biases helps product review systems better predict future product sales. We tested the distribution of product ratings for three product categories (books, DVDs, videos) with data from Amazon collected between February--July 2005: 78%, 73%, and 72% of the product ratings for books, DVDs, and videos are greater or equal to four stars (Figure 1), confirming our proposition that product reviews are overwhelmingly positive. Figure 1 (left graph) shows a J-shaped distribution of all products. This contradicts the law of "large numbers" that would imply a normal distribution. Figure 1 (middle graph) shows the distribution of three randomly-selected products in each category with over 2,000 reviews. The results show that these reviews still have a J-shaped distribution, implying that the J-shaped distribution is not due to a "small number" problem. Figure 1 (right graph) shows that even products with a median average review (around 3-stars) follow the same pattern.

Noise or Quality? Cross-Nested Hierarchical Effects of Culture on Online Ratings

“…The final data set included 1,017 ratings (132 negative ratings and 885 positive ratings) across 520 posts between 786 unique combinations of members with 136 unique post authors' receiving ratings and 153 unique raters, which resulted in 207 unique members in my final sample. The low percentage of negative ratings relative to positive ratings is consistent with previous empirical research (Chevalier & Mayzlin, 2006;Dellarocas & Wood, 2008;Hu et al, 2009), and the regression models that I used to analyze these data do not require the baseline probabilities to be equal (Snijders & Bosker, 2012). Std.…”

“…In all cases, a negative rating is the opposite of a positive rating (i.e., likelihood that a rating will be positive or negative (binary systems) or likelihood that a rating will be closer to the top (positive) or bottom (negative) end of a continuous scale). I frame all hypotheses in relation to negative ratings (see Figure 1) because prior research indicates that negative ratings have a much stronger impact on a variety of outcomes (Chevalier & Mayzlin, 2006) and that negative ratings are much less common (Chevalier & Mayzlin, 2006;Dellarocas & Wood, 2008;Hu, Pavlou, & Zhang, 2009). Eliasoph and Lichterman (2003, p. 737) define a community's culture as "recurrent patterns of interaction that arise from a group's shared assumptions about what constitutes good or adequate participation in the group setting", which they refer to as culture-in-interaction.…”

Emotions and Activity Profiles of Influential Users in Product Reviews Communities

“…This approach has been proved useful in the field of recommender systems [2,25,60,61]. On the other hand, self-selection biases make it difficult to analyze star-rating distributions, as their high bias reduces the heterogeneity of user evaluations, following a J-shaped distribution [26]. This is the case for both EP and DY, where the distribution of star-ratings of FIGURE 6 | The distribution of the total helpfulness (h u ) of users for DY (triangles) and EP (squares).…”

“…This approach has been proved useful in the field of recommender systems [2,25]. On the other hand, self-selection biases difficult the analysis of star-rating distributions, as their high bias reduces the heterogeneity of user evaluations, following a J-shaped distribution [26].…”

StarTrack: The Next Generation (of Product Review Management Tools)

“…As clear from Table 1, both datasets are highly imbalanced, with positive and very positive reviews by far outnumbering mild and negative reviews (this is especially true for TripAdvisor-15763); the fact that the ratings of online consumers tend to be positive was noted e.g., in 12) , and studied in depth in 23) . * 13 …”