Item response theory evaluation of the Light and Spectroscopy Concept Inventory national data set

Abstract

[This paper is part of the Focused Collection on Astronomy Education Research.] This paper presents the first item response theory (IRT) analysis of the national data set on introductory, general education, college-level astronomy teaching using the Light and Spectroscopy Concept Inventory (LSCI). We used the difference between students' pre-and postinstruction IRT-estimated abilities as a measure of learning gain. This analysis provides deeper insights than prior publications both into the LSCI as an instrument and into the effectiveness of teaching and learning in introductory astronomy courses. Our IRT analysis supports the classical test theory findings of prior studies using the LSCI with this population. In particular, we found that students in classes that used active learning strategies at least 25% of the time had average IRTestimated learning gains that were approximately 1 logit larger than students in classes that spent less time on active learning strategies. We also found that instructors who want their classes to achieve an improvement in abilities of average Δθ ¼ 1 logit must spend at least 25% of class time on active learning strategies. However, our analysis also powerfully illustrates the lack of insight into student learning that is revealed by looking at a single measure of learning gain, such as average Δθ. Educators and researchers should also examine the distributions of students' abilities pre-and postinstruction in order to understand how many students actually achieved an improvement in their abilities and whether or not a majority of students have moved to postabilities significantly greater than the national average.

Balancing content of computerized adaptive testing for the Force Concept Inventory

“…The FCI-CAT can be validated and improved in means other than content balancing as well, for example, in terms of test security [11], removing gender unfair items [22], or by dropping one item from each locally dependent pair [23]. These concerns are important aspects to consider when thinking about instrument validity.…”

Development of the Water Instrument: a comprehensive measure of students’ knowledge of fundamental concepts in general chemistry

“…One assumption of using an IRT approach is unidimensionality. Support for unidimensionality was provided by the acceptable model fit indices (Wallace et al, 2017), which will be presented in detail in the Results section.…”

Item response curve analysis of Likert scale on learning attitudes towards physics

“…Certain computer programs are required to complete the modeling part of IRT analysis. IRT was applied not only to evaluate test items [26,30,31,35,36] but also to measure student learning [37,38].…”