Scoring best-worst data in unbalanced many-item designs, with applications to crowdsourcing semantic judgments

Abstract

Best-worst scaling is a judgment format in which participants are presented with a set of items and have to choose the superior and inferior items in the set. Best-worst scaling generates a large quantity of information per judgment because each judgment allows for inferences about the rank value of all unjudged items. This property of best-worst scaling makes it a promising judgment format for research in psychology and natural language processing concerned with estimating the semantic properties of tens of thousands of words. A variety of different scoring algorithms have been devised in the previous literature on best-worst scaling. However, due to problems of computational efficiency, these scoring algorithms cannot be applied efficiently to cases in which thousands of items need to be scored. New algorithms are presented here for converting responses from best-worst scaling into item scores for thousands of items (many-item scoring problems). These scoring algorithms are validated through simulation and empirical experiments, and considerations related to noise, the underlying distribution of true values, and trial design are identified that can affect the relative quality of the derived item scores. The newly introduced scoring algorithms consistently outperformed scoring algorithms used in the previous literature on scoring many-item best-worst data.

When is best-worst best? A comparison of best-worst scaling, numeric estimation, and rating scales for collection of semantic norms

“…This process was repeated 128 times to create the master stimulus set over which participants would make best-worst judgments. Four-tuples were created according to the design suggestions laid out by Hollis (2017): (1) no two words appeared together in more than one 4-tuple, (2) every word appeared in an equal number of 4-tuples, and (3) no 4-tuple was viewed more than once across all participants. These features each increase the amount of information gained per judgment made.…”

“…Scoring best-worst data A detailed explanation of the various ways to convert best-worst data into estimates of latent dimensions (i.e., scoring methods), along with their respective benefits and shortcomings, is provided by Hollis (2017). For the current analysis, each of the five scoring methods described in Hollis (2017) were applied to each of the four latent dimensions being estimated.…”

“…Very little work has been done in finding ways to reduce measurement error of these methodologies in a crowdsourcing environment, or alternatively of devising different ways of collecting human judgments. One exception is the work of Hollis (2017), who developed scoring methods for a response format called best-worst scaling. In a series of simulation experiments conducted by Hollis (2017), best-worst scaling showed promise for being a more efficient response format than rating scales for collecting human semantic judgments (see also, Kiritchenko & Mohammad, 2016.…”

“…The primary contributions of Hollis (2017) were in identifying and assessing methodological considerations for using best-worst scaling. This included how to construct sets of N items to provide maximal information per decision, and how to use results of best-worst trials to place items along an ordinal scale that is informative of the underlying latent dimension that judgments are being made over.…”

Big

“…12 Gold questions were annotated more than eight times. 13 More complex optimization algorithms exist, such as those described in (Hollis, 2018); however, our past experiments showed that the simple counting procedure obtained the most reliable results.…”