An item response theory approach to longitudinal analysis with application to summer setback in preschool language/literacy

Abstract

Background: As the popularity of classroom observations has increased, they have been implemented in many longitudinal studies with large probability samples. Given the complexity of longitudinal measurements, there is a need for tools to investigate both growth and the properties of the measurement scale.Methods: A practical IRT model with an embedded growth model is illustrated to examine the psychometric characteristics of classroom assessments for preschool children, and also to show how nonlinear learning over time can be investigated. This approach is applied to data collected for the Academic Rating Scale (ARS) in the literacy domain, which was administered on four occasions over two years. Results:The model enabled an effective illustration of overall and individual gains over two academic years. In particular, a significant de-acceleration in latent literacy skills during summer was observed. The results also provided psychometric support for the argument that ARS literacy can be used to assess developmental skill levels consistent with theories of early literacy acquisition. Conclusions: The proposed IRT approach provided growth parameters that are estimated directly, rather than obtaining these coefficients from estimated growth scores-which may result in biased and inconsistent estimates of growth parameters. The model is also capable of simultaneously representing parameters of items and persons.

Application of a Longitudinal IRTree Model: Response Style Changes Over Time

“…To evaluate the longitudinal assumption that item parameters were invariant across the two waves of data collection, we undertook a systematic procedure using the same approach as Kim and Camilli (2014). To evaluate longitudinal invariance from an IRT Millsap (2010) suggests fitting a model that constrains the item parameters to be invariant across all measurement occasions.…”

“…Then, compare, using a likelihood ratio test, the invariant model to one that allows for varying item parameters across measurement occasions. This is essentially the approach we have taken, as have Kim and Camilli (2014), but adapted to a Bayesian framework.…”

“…The longitudinal model for response processes builds on the longitudinal IRT model for dichotomous responses (Kim & Camilli, 2014), which leads into an extension for response styles, the longitudinal IRTree (LIRTree; Ames & Leventhal, 2021).…”

“…The population parameter effects of individuals are µ t and normalθ i t , where µ t is average performance at time t across all i individuals and normalθ i t is the specific effect for individual i as a departure from µ t . This model is multidimensional because each of the t time points provides a different measurement to the same, constant instrument (Kim & Camilli, 2014). The covariance structure of θ can be structured or unstructured.…”

Chronicle of a Measurement Unforetold

“…IRT analyses also define a set of functions such as the item characteristic function, the item information function, and the test information function, which provide an in-depth understanding of the scale and the items. While applications of IRT for longitudinal and intensive longitudinal settings are scarce (e.g., Cai, 2010;Hecht et al, 2019;Kim & Camilli, 2014;Rijn et al, 2010), developing IRT models for intensive longitudinal settings is a promising endeavour that can contribute to improve the measurement of psychological dynamics, because it can inform about the performance and quality of the items and scales used in intensive longitudinal research.…”