Best-worst scaling improves measurement of first impressions

Abstract

A common goal in psychological research is the measurement of subjective impressions, such as first impressions of faces. These impressions are commonly measured using Likert ratings. Although these ratings are simple to administer, they are associated with response issues that can limit reliability. Here we examine best-worst scaling (BWS), a forced-choice method, as a potential alternative to Likert ratings for measuring participants’ facial first impressions. We find that at the group level, BWS scores correlated almost perfectly with Likert scores, indicating that the two methods measure the same impressions. However, at the individual participant level BWS outperforms Likert ratings, both in terms of ability to predict preferences in a third task, and in terms of test-retest reliability. These benefits highlight the power of BWS, particularly for use in individual differences research.

Beyond Likert ratings: Improving the robustness of developmental research measurement using best–worst scaling

“…However, the same task could also be designed using other software packages, e.g., the supportBWS package for R (Aizaki et al, 2014), and presented in any online or paper medium. See Additional Files associated with Burton et al, 2019 for helpful materials including "A beginners guide to the process of designing, running and scoring a Best-Worst Scaling Task', annotated R scripts and supporting files.…”

“…We also tested a group of adult participants on the same tasks. Based on the findings of Burton et al (2019) and Kiritchenko and Mohammad (2017), we also expected to find the same pattern of improved individual and group-level reliability for this adult group.…”

“…A power analysis was carried out using the pwr package in R Version 3.5 (R Core Team, 2013). This analysis was based on the results of a previous study conducted by Burton et al (2019). These authors observed a difference of d = 0.68 between the self-consistency correlations in their adult BWS (r = .66) and Likert (r = .53) groups for ratings of attractiveness, and d = .73 between adult BWS (r = .76) and Likert (r = .63) groups for ratings of trustworthiness.…”

“…BWS has already been demonstrated to improve reliability for adult respondents compared to Likert ratings. When judging the attractiveness or distinctiveness of faces (Burton et al, 2019), and when judging verbal statements, (Kiritchenko & Mohammad, 2017), BWS produces both more reliable grouplevel ratings and higher consistency in individual participants' ratings across sessions than the Likert method. The studies noted here made use of crowdsourcing platforms for their recruitment: across three studies, Burton et al (2019) tested 924 Amazon Mechanical Turk workers (431 male) who were Caucasian and resided in the USA, with a mean age of 36.2 years; Kiritchenko and Mohammad (2017) used CrowdFlower, noting only that their workers were required to be native English speakers from the USA.…”

To Rate or Not To Rate: Investigating Evaluation Methods for Generated Co-Speech Gestures

“…In relation to biomedical image assessments, where evaluation considers the visual quality of the stimuli, Phelps et al [27] found that pairwise comparisons and ranked Likert scores made for more accurate assessments in comparison to the use of non-ranked Likert scores. Burton et al [2] compared rating scales with best-worst scaling, another variant of the rank-based approach. In this study, participants were asked to select the most attractive and least attractive faces in a series of images.…”