The Effect of Rating Scale Length on the Occurrence of Inappropriate Category Use for the Assessment of Job Satisfaction: an Experimental Online Study

Abstract

When job satisfaction is measured in national panel surveys using a rating scale that consists of many response categories the psychometric quality of the data obtained is often reduced. One reason lies in an inappropriate category use (e.g., in terms of response styles or ignoring superfluous categories), which occurs when respondents are faced with an overwhelmingly large number of response options. The use of response styles can also be triggered by stable respondent characteristics. The objective of the present between-subject experimental study is to explore the impact of rating scale length on the occurrence of inappropriate category use and scale reliability. In addition, this study investigates which stable respondent characteristics and job-related factors consistently predict the use of a particular response style across all experimental conditions. A sample of MTurk workers (N = 7042) filled out a 12-item online questionnaire on aspects of job satisfaction, with a 4-, 6-, or 11-point rating scale randomly assigned. Considering the three-dimensional structure of the job satisfaction measure, we applied a multidimensional extension of the restricted mixed generalized partial credit model to explore category use patterns within each condition. The results show a similar configuration of three response-style classes in all conditions. Nevertheless, the proportion of respondents who used the rating scale inappropriately was lower in the conditions with fewer response categories. An exception was the extreme response style, which showed a similar prevalence rate in all conditions. Furthermore, we found that the use of extreme response style can be explained by a high level of general self-efficacy and perceived job autonomy, regardless of rating scale length. The findings of the study demonstrate that the prevalence of inappropriate category use can be reduced by administering rating scales with six or four response categories instead of eleven. These findings may be extended to other domains of life satisfaction.

Comparing Irt Properties of Different Likert Scale Lengths: a Case From Attitudinal Measurement on Physics Education Research

“…However, the 11-point has a wider spectrum of latent abilities than the 5-point scale. These results are relevant to previous studies by Kutscher and Eid (2020) and Weng (2004). Kutscher and Eid (2020) were in the same position as the author in articulating the meaning of the peak of the TIF curve.…”

“…These results are relevant to previous studies by Kutscher and Eid (2020) and Weng (2004). Kutscher and Eid (2020) were in the same position as the author in articulating the meaning of the peak of the TIF curve. In this part of the curve, there is the highest probability of a latent continuum that potentially be endorsed by the rater/ respondents.…”

“…Admittedly, the inquiries to this subject have attracted several psychometrical scholars even dating back 70 years ago (Bendig, 1953;Miller, 1956). Moreover, to date, there are several investigations approached by the modern measurement theory such as item response theory (IRT) as revealed by Kutscher and Eid (2020) as well Lee and Paek (2014). Unfortunately, the research community still has no consensus about the optimal scale length that should be adapted in implementing the Likert rating scale.…”

“…Methodologically, the performance behavior of different Likert scale lengths can be studied using two approaches either through experimentation (Donnellan and Rakhshani, 2020;Kutscher and Eid, 2020;Shi et al, 2021) or using computer simulation (Goggin and Stoker, 2014;Sangthong, 2020). Experimental study has several advantages that empirically consider the psychological aspects of respondents that were unable to be explained by the simulation study.…”

“…Previous works reported that there were several statistical measures that have been examined to evaluate the experimental setting of different Likert scale lengths. They were reliability (Donnellan and Rakhshani, 2020), discrimination index (Alan and Kabasakal, 2020;Daher et al, 2015;Goggin and Stoker, 2014;Freihat, 2019;Garner, 1960;Weng, 2004), criterion validity (Simms et al, 2019), factor analysis (Shi et al, 2021), inferential statistics (Sangthong, 2020), polytomous item response theory (Kutscher and Eid, 2020), Rasch model (Daher et al, 2015;Ilhan and Guler, 2017), structural equation modeling (Maydeu-Olivares et al, 2009), and information theory (Green and Rao, 1970;Miller, 1956;Parducci and Wedell, 1986) (Hambleton et al, 1991;Harwell and Gatti, 2001;Lee and Paek, 2014).…”