Do Concept Inventories Actually Measure Anything?

Abstract

Although concept inventories are among the most frequently used tools in the physics and astronomy education communities, they are rarely evaluated using item response theory ͑IRT͒. When IRT models fit the data, they offer sample-independent estimates of item and person parameters. IRT may also provide a way to measure students' learning gains that circumvents some known issues with Hake's normalized gain. In this paper, we review the essentials of IRT while simultaneously applying it to the Star Properties Concept Inventory. We also use IRT to explore an important psychometrics debate that has received too little attention from physics and astronomy education researchers: What do we mean when we say we "measure" a mental process? This question leads us to use IRT to address the provocative question that constitutes the title of this paper: Do concept inventories actually measure anything?

Applicability of the Newtonian gravity concept inventory to introductory college physics classes

“…13,37 Finally, item discrimination is calculated with the point-biserial correlation between student performance on the item and their overall performance on the NGCI. 25,34 For item discrimination, anything greater than 0.30 indicates that students' scores on that item are well-correlated with their total scores. 13,25 Additional information from student response patterns and interviews is critical for providing context to these statistics and to further the discussion of the validity of the NGCI.…”

“…Values over 0.70 are conventionally accepted as being internally reliable. 25,36 Item difficulty is calculated as the percentage of students who answered incorrectly, such n/a n/a that a very high percentage indicates that most students answered incorrectly (i.e., the item may be too difficult) and a very low percentage indicates that most students answered correctly (i.e., the item may be too easy). In order for a survey to be of appropriate difficulty, most items should have difficulty values between 0.20 and 0.80.…”

“…6,[11][12][13]25 Three survey iterations resulted in the full 26-item Newtonian Gravity Concept Inventory (NGCI). Classical Test Theory statistics show that the NGCI has high reliability (Cronbach's Alpha ¼ 0.84) and has appropriate item difficulty and discrimination.…”

Measuring Learning Gains in Chemical Education: A Comparison of Two Methods

“…A recent study in astronomy 37 has pointed out the weaknesses of measures based on classical test theory, normalized learning gain calculations, and the advantages of Rasch-based calculations. In the present work, the Rasch learning gain, RLG, will be calculated in the same manner (eq 5) used by Wallace and Bailey.…”

“…In the present work, the Rasch learning gain, RLG, will be calculated in the same manner (eq 5) used by Wallace and Bailey. 37 This difference calculation is using the Rasch estimate of ability, and not the raw score. This is acceptable here because the Rasch ability estimates represent true linear measures and are suitable for use in a difference calculation 23 where raw scores are not.…”