When is best-worst best? A comparison of best-worst scaling, numeric estimation, and rating scales for collection of semantic norms

Hollis, Geoff; Westbury, Chris

doi:10.3758/s13428-017-1009-0

Cited by 38 publications

(36 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Third, SpAM could be compared to other emerging techniques for measuring similarity and producing latent representational spaces. In Best-worst scaling (BWS), for example, participants are given a set of N (usually four but perhaps optimally six; Hollis, 2019) items and are asked to pick the most superior and most inferior items along some dimension (Louviere, Flynn, & Marley, 2015;Hollis, 2018;Hollis & Westbury, 2018). Such a trial is efficient in that these two overt choices implicitly produce 2N-3 ordinal comparisons (which can be converted to continuous scores on the relevant dimension with various scoring algorithms; Hollis, 2018).…”

Section: Future Directionsmentioning

confidence: 99%

The spatial arrangement method of measuring similarity can capture high-dimensional semantic structures

et al. 2020

View full text Add to dashboard Cite

Psychologists collect similarity data to study a variety of phenomena including categorization, generalization and discrimination, and representation itself. However, collecting similarity judgments between all pairs of items in a set is expensive, spurring development of techniques like the Spatial Arrangement Method (SpAM; Goldstone, Behavior Research Methods, Instruments, & Computers, 26, 381-386, 1994), wherein participants place items on a two-dimensional plane such that proximity reflects perceived similarity. While SpAM greatly hastens similarity measurement, and has been successfully used for lower-dimensional, perceptual stimuli, its suitability for higher-dimensional, conceptual stimuli is less understood. In study 1, we evaluated the ability of SpAM to capture the semantic structure of eight different categories composed of 20-30 words each. First, SpAM distances correlated strongly (r = .71) with pairwise similarity judgments, although below SpAM and pairwise judgment splithalf reliabilities (r's > .9). Second, a cross-validation exercise with multidimensional scaling fits at increasing latent dimensionalities suggested that aggregated SpAM data favored higher (> 2) dimensional solutions for seven of the eight categories explored here. Third, split-half reliability of SpAM dissimilarities was high (Pearson r = .90), while the average correlation between pairs of participants was low (r = .15), suggesting that when different participants focus on different pairs of stimulus dimensions, reliable high-dimensional aggregate similarity data is recoverable. In study 2, we show that SpAM can recover the Big Five factor space of personality trait adjectives, and that cross-validation favors a four-or five-dimension solution on this dataset. We conclude that SpAM is an accurate and reliable method of measuring similarity for high-dimensional items like words. We publicly release our data for researchers.

show abstract

Section: Future Directionsmentioning

confidence: 99%

The spatial arrangement method of measuring similarity can capture high-dimensional semantic structures

et al. 2020

View full text Add to dashboard Cite

show abstract

“…BWS may also be a fruitful method for other research areas that use similar materials: for instance, object perception or body perception. More generally, given our findings and those from research in the areas of language processing (Hollis, 2018; Hollis & Westbury, 2018) and personality (Lee et al, 2007, 2008), we recommend that researchers in any area of experimental psychology should consider whether BWS might be an appropriate substitute for Likert ratings, especially in individual differences research where reliability is particularly critical.…”

Section: Discussionmentioning

confidence: 78%

“…perceived age). Because each BWS trial includes multiple faces, it may be easier for participants to maintain a single, consistent interpretation of the trait to be rated in the BWS paradigm, rather than varying their interpretation stimulus-by-stimulus (Hollis & Westbury, 2018). Following this line of reasoning, complex trait judgements that involve synthesising several cues might benefit more from the BWS method.…”

Section: Discussionmentioning

confidence: 99%

“…The more attractive a face is to the participant, the more trials on which it will be selected as “most attractive” and the fewer trials on which it will be selected as “least attractive”. The rankings can be estimated either with simple scoring algorithms (see Hollis, 2018; Hollis & Westbury, 2018) or by estimating regression models (Louviere, Flynn, & Marley, 2015).

Fig.…”

Section: Best-worst Scaling As An Alternative To Likert Ratings In Fimentioning

confidence: 99%

“…However, the use of BWS in experimental psychology has so far been limited. The existing applications in the psychology literature concern only verbal materials - either for establishing semantic norms (Hollis, 2018; Hollis & Westbury, 2018) or for ranking value statements in the personality literature (Lee, Soutar, & Louviere, 2007, 2008). To the best of our knowledge BWS has not yet been investigated as a potential method in the study of visual perception, and in particular has not been used to measure facial first impressions.…”

Section: Best-worst Scaling As An Alternative To Likert Ratings In Fimentioning

confidence: 99%

See 2 more Smart Citations

Best-worst scaling improves measurement of first impressions

Burton

Rigby

et al. 2019

Cogn. Research

View full text Add to dashboard Cite

A common goal in psychological research is the measurement of subjective impressions, such as first impressions of faces. These impressions are commonly measured using Likert ratings. Although these ratings are simple to administer, they are associated with response issues that can limit reliability. Here we examine best-worst scaling (BWS), a forced-choice method, as a potential alternative to Likert ratings for measuring participants’ facial first impressions. We find that at the group level, BWS scores correlated almost perfectly with Likert scores, indicating that the two methods measure the same impressions. However, at the individual participant level BWS outperforms Likert ratings, both in terms of ability to predict preferences in a third task, and in terms of test-retest reliability. These benefits highlight the power of BWS, particularly for use in individual differences research.

show abstract