Breaking free from the limitations of classical test theory: Developing and measuring information systems scales using item response theory

Abstract

Information systems (IS) research frequently uses survey data to measure the interplay between technological systems and human beings. Researchers have developed sophisticated procedures to build and validate multi-item scales that measure latent constructs. The vast majority of IS studies uses classical test theory (CTT), but this approach suffers from three major theoretical shortcomings: (1) it assumes a linear relationship between the latent variable and observed scores, which rarely represents the empirical reality of behavioral constructs; (2) the true score can either not be estimated directly or only by making assumptions that are difficult to be met; and (3) parameters such as reliability, discrimination, location, or factor loadings depend on the sample being used. To address these issues, we present item response theory (IRT) as a collection of viable alternatives for measuring continuous latent variables by means of categorical indicators (i.e., measurement variables). IRT offers several advantages: (1) it assumes nonlinear relationships; (2) it allows more appropriate estimation of the true score; (3) it can estimate item parameters independently of the sample being used; (4) it allows the researcher to select items that are in accordance with a desired model; and (5) it applies and generalizes concepts such as reliability and internal consistency, and thus allows researchers to derive more information about the measurement process. We use a CTT approach as well as Rasch models (a special class of IRT models) to demonstrate how a scale for measuring hedonic aspects of websites is developed under both approaches. The results illustrate how IRT can be successfully applied in IS research and provide better scale results than CTT. We conclude by explaining the most appropriate circumstances for applying IRT, as well as the limitations of IRT.

Developing a Short-Form Version of the HIV Disability Questionnaire (SF-HDQ) for Use in Clinical Practice: A Rasch Analysis

“…Rasch Analysis: We conducted a Rasch analysis, a popular method for developing PROMs, assessing cross-cultural validity, and guiding item reduction [31,[34][35][36][37]. The Rasch model uses a logistic function to indicate the probability that an individual responds to a particular item (response option) and is dependent on both the individual ability and the di culty (or severity) of the item [38].…”

“…As the items are hierarchically ordered, Rasch analysis allows an individual's location on the continuum to be precisely estimated using few items, making it an ideal approach for item reduction [34][35][36][37]. For the development of the SF-HDQ, we conducted a Rasch analysis using a rating scale model [39] focused on the severity scale (0-4 response options).…”

Measuring Latent Constructs in Nonprofit Surveys with Item Response Theory: The Example of Political Ideology

“…Originating in educational testing, a "difficult" question will function well among those more able. Here, it can also be conceptualized as the item's "location on the latent trait" (Rusch et al 2016). Discrimination, on the other hand, is how well the each item differentiates between conservative and liberal ideologies, and the steeper the curve, the better the item is able to discriminate between a liberal voter and a conservative voter (Baker 2001).…”

The Factors Affecting IT Insourcing in the Financial and Oil & Gas Industry in Malaysia

“…Hence, CFI RMSEA and SRMR were referenced for model fitness (Hair, et al, 2010). Hu and Bentler (1999) argued that the higher CFI value .950 indicates better fit of the model however, several other scholars like Bagozzi and Yi (1988), Baumgartner and Homburg (1996), Diamantopoulos and Siguaw (2000) and Ping (2004) indicated CFI indices of .900 was good fit (Rusch, et al, 2017;Vieira, 2011).…”