The design space of opinion measurement interfaces

Abstract

Rating interfaces are widely used on the Internet to elicit people"s opinions. Little is known, however, about the effectiveness of these interfaces and their design space is relatively unexplored. We provide a taxonomy for the design space by identifying two axes: Measurement Scale for absolute rating vs. relative ranking, and Recall Support for the amount of information provided about previously recorded opinions. We present an exploration of the design space through iterative prototyping of three alternative interfaces and their evaluation. Among many findings, the study showed that users do take advantage of recall support in interfaces, preferring those that provide it. Moreover, we found that designing ranking systems is challenging; there may be a mismatch between a ranking interface that forces people to specify a total ordering for a set of items, and their mental model that some items are not directly comparable to each other.

Choice Architecture for Human-Computer Interaction

“…Aiming at defining a general model of rating scales, we examined thirteen rating scales described by van Barneveld & van Setten (2004), Gena et al (2011) and Nobarany et al (2012): 3-, 5-and 10-point stars, bare numbers, smileys, sliders ranging -10/+10, -1/+1 and 0/10, likert-like scales ranging -10/+10 and 1/5 and 1-, 2-, and 3-point thumbs (see Figure 1). First, we identified a list of features which could be used to describe them, based on the literature and our insights, and we organized it in a model (Section 2.1).…”

“…Then, observing how these features tend to combine in the examined scales, we identified three general classes of rating scales (Section 2.2). van Barneveld & van Setten (2004), Nobarany et al (2012) described rating scales through different features, as reported in Table 1.…”

“…[Van Barnevald and Van Setten 2004], [Nobarany et al 2012] and Then, we grouped rating scale features by means of card sorting. Card sorting is a quick and popular method for structuring information, often used in information architecture and user-centred design.…”

“…Cosley et al (2003) studied granularity, numeric labels and visual metaphors in a series of user studies, concluding that designers can allow users to choose the scales they prefer, since they found no significant difficulties in managing multiple scales. Nobarany et al (2012) focused on the measurement scale, distinguishing between "absolute rating" and "relative ranking", and on the recall support, i.e., the ability to offer information about previously provided input. They concluded that users take advantage of recall support, while ranking interfaces may be quite challenging since they force users to specify a total ordering for items.…”

“…Rating scales are widely studied in the literature, especially in survey design (Garland, 1991;Colman et al, 1997;Amoo & Friedman, 2001;Dawes, 2008) and Human-Computer DOI: 10.4018/ijthi.2015010103 Interaction (HCI) (Cosey et al, 2003;van Barneveld & van Setten, 2004;Nobarany et al, 2012;Herlocker et al, 2004), but not in terms of preferential choices, i.e. not focusing on users' decision making process.…”