The World Beyond Rating Scales

Abstract

Researchers constructing a new questionnaire think very carefully about a lot of things: the construct definition, the target population, the wording of the items, the item selection, and so on. These are all explicit steps in the test construction process as it is described by textbooks and research articles (Clark & Watson, 1995;Simms, 2008;Ziegler, 2014). One aspect that appears to receive less attention is the choice of response format. When the questionnaire is a self-report (or other-report) measure of one or more psychological constructs, test constructors appear to automatically implement a rating scale such as strongly disagree to strongly agree as the response format. The reason for this is that rating scales have served us well in the past and continue to do so.However, in this editorial we will argue that choosing the response format should be an explicit step in the test construction process that deserves special attention and considerable thought. In fact, the response format should be chosen to fit the construct best. We will also argue that we need a greater diversity of response formats and more research on them.

Comparing the validity of trait estimates from the multidimensional forced-choice format and the rating scale format.

“…At the most general level, our findings imply that the response format matters and that we cannot assume that different response formats are interchangeable and yield equally good assessments and therefore diagnostic inferences in practice (see also Wetzel & Greiff, 2018).…”

“…Thus, future studies could also take into account other response formats and not only compare the MFC format to the RS format (for an overview on response formats in self-report questionnaires see Wetzel & Greiff, 2018). In addition, it would be interesting to investigate what drives the change in construct between different response formats.…”

Comparison of Response Scales as Measures of Indoor Environmental Perception in Combined Thermal and Acoustic Conditions

“…The limited information on response scales for combined environmental assessments is a challenge to the success of subjective building performance. To the best of the authors' knowledge, no studies on an optimized response format for combined environmental assessments have been reported to date, although the choice of response format should be an explicit step in the process of constructing a questionnaire, and different response formats may be appropriate for different constructs [40].…”

Effects of Response Format on Psychometric Properties and Fairness of a Matrices Test: Multiple Choice vs. Free Response

“…The question regarding the most appropriate response format to gain valid information concerning test takers' characteristics is currently becoming a "hot topic" within the assessment of personality in the job context with reference to the newly emerged forced choice format (see Usami et al, 2016;Lin and Brown, 2017;O'Neill et al, 2017;Brown and Maydeu-Olivares, 2018;Wetzel and Greiff, 2018). However, studies addressing response formats within the domain of the assessment of knowledge, skills, and abilities with stem-equivalent items are rare.…”