Rating Norms Should be Calculated from Cumulative Link Mixed Effects Models

Abstract

Studies which provide norms of Likert ratings typically report per-item summary statistics. Traditionally, these summary statistics comprise the mean and the standard deviation (SD) of the ratings, and the number of observations. Such summary statistics can preserve the rank order of items, but provide distorted estimates of the relative distances between items because of the ordinal nature of Likert ratings. Inter-item relations in such ordinal scales can be more appropriately modelled by cumulative-link mixed effects models (CLMMs). In a series of simulations, and with a reanalysis of an existing rating norms dataset, we show that CLMMs can be used to more accurately norm items, and can provide summary statistics analogous to the traditionally reported means and SDs, but which are disentangled from participants’ response biases. CLMMs can be applied to solve important statistical issues that exist for more traditional analyses of rating norms.

Emoji Identification and Emoji Effects on Sentence Emotionality in ASD-Diagnosed Adults and Neurotypical Controls

“…Similarly, it could be argued that the non-difference between participant groups in Study 1 'angry' data (p = 0.053) is problematic and potentially influenced by participant execution errors and/or 'lucky guesses'. However, this seems unlikely given the robustness of the analysis (e.g., Taylor et al, 2021). Study 2 'message rating' data could also have been influenced by participant execution error; however, this also seems very unlikely.…”

“…Most prior work in this area has used general linear model-type analyses, despite outcomes being measured ordinally, and/or the assumptions of the analyses being violated. Likert scales are ordinal by nature; although scale points might appear equally spaced and equivalent, there is no evidence that every participant agrees as to what constitutes a response at each scale point, or that each participant's evaluations of adjacent points are equal (Taylor et al, 2021). The relationship between participant responses and underlying latent dimension(s) are at-best underspecified (Taylor et al, 2021).…”

“…Likert scales are ordinal by nature; although scale points might appear equally spaced and equivalent, there is no evidence that every participant agrees as to what constitutes a response at each scale point, or that each participant's evaluations of adjacent points are equal (Taylor et al, 2021). The relationship between participant responses and underlying latent dimension(s) are at-best underspecified (Taylor et al, 2021). The analytical approach used in Study 2-cumulative link mixed modelling (CLMM)-maps ordinal outcomes against ordered regions of a latent distribution (Bürkner & Vuorre, 2019;McCullagh, 1980).…”

“…The analytical approach used in Study 2-cumulative link mixed modelling (CLMM)-maps ordinal outcomes against ordered regions of a latent distribution (Bürkner & Vuorre, 2019;McCullagh, 1980). We utilised this approach to estimate the fixed effects (participant group, emoji type, group × emoji type), embracing the 'randomness' generated by individual participants and items included in the study (Taylor et al, 2021). There is clear evidence of the problems in using general linear modelling/ANOVA to evaluate ordinal data sets, and the need for CLMM approaches within experimental studies has been successfully argued by, for example, Liddell and Kruschke (2018).…”

“…We adjusted our analyses to account for multiple comparisons, particularly in Study 1. We performed state-of-the-art analyses (Taylor et al, 2021), rather than falling back on non-optimal analyses such as those based around general linear models or nonparametric tests. It must be recognised, however, that in many cases our effect sizes are not particularly large, either in statistical or 'real' terms.…”