Addressing the elephant in the middle: Implications of the midscale disagreement problem through the lens of Body-Object Interaction ratings

Abstract

When participants disagree about their judgments on a Likert-type scale, the average rating will be naturally drawn towards its middle. The present work’s goal is to explore the implications of this midscale disagreement problem for psycholinguistic norms by using the literature on Body-Object Interaction (BOI) ratings as a case study. Through a series of graphical analyses, we argue that (i) the average rating of most midscale items cannot be interpreted as their true position on the variable’s continuum; (ii) other variables driving the disagreement in judgements can introduce an independent midscale effect in word processing performances; (iii) the typical sample sizes used by norming studies are likely insufficient to reliably detect disagreements and can lead to significant measurement error. A methodological review of studies on BOI’s effect in word processing reveals that most of them suffer from the midscale disagreement problem, either because of inadequate word sampling or statistical modelling. Whereas these observations provide initial clues for the interpretation and use of the ratings, it remains difficult to determine the full scope of the disagreement problem based only on the summary statistics reported by rating studies. To address this point, we present new BOI ratings for a set of 1019 French words which we use to perform item-level descriptive and exploratory analyses. Overall, the results confirm that unipolar Likert-type scale ratings such as BOI capture the dimension of interest mainly at the two ends of the scale, while they represent increasing disagreement among participants as they approach the middle. These observations provide initial best-practice recommendations for the use and interpretation of subjective variables. Our analyses can additionally serve as general guidelines to interpret similar ratings and to assess the validity of previous findings in the literature based on standard summary statics.