Navigating by the Stars: Investigating the Actual and Perceived Validity of Online User Ratings

Abstract

This research documents a substantial disconnect between the objective quality information that online user ratings actually convey and the extent to which consumers trust them as indicators of objective quality. Analyses of a data set covering 1272 products across 120 vertically differentiated product categories reveal that average user ratings (1) lack convergence with Consumer Reports scores, the most commonly used measure of objective quality in the consumer behavior literature, (2) are often based on insufficient sample sizes which limits their informativeness, (3) do not predict resale prices in the used-product marketplace, and (4) are higher for more expensive products and premium brands, controlling for Consumer Reports scores. However, when forming quality inferences and purchase intentions, consumers heavily weight the average rating compared to other cues for quality like price and the number of ratings. They also fail to moderate their reliance on the average user rating as a function of sample size sufficiency. Consumers’ trust in the average user rating as a cue for objective quality appears to be based on an “illusion of validity.”

The Economic and Behavioral Consequences of Online User Reviews

“…Experts' opinions, though, are not as ubiquitous as online reviews. Further, consumers relate more to the opinions of peers, rather than those of professionals – who are paid to write reviews (De Langhe et al ., 2016).…”

“…Alongside online reviews, Öğüt and Onur Taş (2012) included a more objective quality cue, that is, the number of stars assigned by national agencies to hotels. Using data from Booking, one of the most popular OTAs, the author found that an increase in valence (average rating) has a positive and statistically significant effect on hotel room sales (proxied by the number of reviews); the official star rating, instead, does not affect the hotel performance – providing support to studies showing that consumers do read official certification labels but eventually put more weight on peers' feedback (De Langhe et al ., 2016). The positive role of valence has also been established in other industries.…”

“…A 2009 research report by McKinsey 17 challenged this traditional view by theorizing a circular process called the “consumer decision journey,” Here, the consumer may add or subtract brands at any given time during the awareness and consideration phases; being exposed to constant advertisements, recommendations and other marketing‐related activities on digital devices have made the funnel metaphor obsolete. Further, in today's landscape, consumers are empowered agents, as they can share their opinions with a large audience and become influential spokespersons, through rating systems, forums, and social networks (De Langhe et al ., 2016). This means that, after the purchase, they become active users evaluating their experience with the product and, possibly, informing others.…”

“…It has been shown that low‐motivation consumers (i.e., those who spend little time in searching) tend to exclusively rely on heuristic cues when making a purchase and they might make suboptimal choices (Gupta and Harris, 2010). Furthermore, consumers tend to “place enormous weight on the average user rating as an indicator of objective quality compared to other cues” (De Langhe et al ., 2016, p. 819), such as official certifications and expert reviews. Nevertheless, consumers are widely and constantly exposed to reviews of prior consumers, even through solicitation systems aims at adding volume to review count (Litvin and Sobel, 2019).…”

Visual comparison of two data sets: Do people use the means and the variability?

“…Even in adulthood, we are often presented with mean values in newspapers or television adverts (e.g., comparing the miles per gallon of two car models or the battery life of two smartphones) and expected to decide whether there is a meaningful difference. Problematically, research suggests that we place more weight than we should on these types of average ratings (e.g., de Langhe, Fernbach, & Lichtenstein, 2016). Statistically, information about the means alone is insufficient for making such decisions.…”