Multidimensional item response theory and the Force Concept Inventory

Abstract

Research on the test structure of the Force Concept Inventory (FCI) has largely been performed with exploratory methods such as factor analysis and cluster analysis. Multidimensional Item Response Theory (MIRT) provides an alternative to traditional exploratory factor analysis which allows statistical testing to identify the optimal number of factors. Application of MIRT to a sample of N ¼ 4716 FCI post-tests identified a 9-factor solution as optimal. Additional analysis showed that a substantial part of the identified factor structure resulted from the practice of using problem blocks and from pairs of similar questions. Applying MIRT to a reduced set of FCI items removing blocked items and repeated items produced a 6-factor solution; however, the factors still had little relation the general structure of Newtonian mechanics. A theoretical model of the FCI was constructed from expert solutions and fit to the FCI by constraining the MIRT parameter matrix to the theoretical model. Variations on the theoretical model were then explored to identify an optimal model. The optimal model supported the differentiation of Newton's 1st and 2nd law; of one-dimensional and three-dimensional kinematics; and of the principle of the addition of forces from Newton's 2nd law. The model suggested by the authors of the FCI was also fit; the optimal MIRT model was statistically superior.

Extending modified module analysis to include correct responses: Analysis of the Force Concept Inventory

“…The incorrect communities identified were generally consistent with those identified by MMA. The correct communities generally followed those identified by Stewart et al [16]; therefore, MMA-P productively identified both correct solution structure and incorrect structure at the same time. MMA-P also identified communities which mixed correct and incorrect answers.…”

Exploring the structure of misconceptions in the Force and Motion Conceptual Evaluation with modified module analysis

“…The optimal model presented in Study 2 and a similar study of the FCI [12] provide further evidence for the difference in coverage of the two instruments with the optimal model of the FCI requiring 19 principles (fundamental reasoning steps) while the optimal model of the FMCE required only 8 principles. The two instruments also differed starkly in their re-use of principles with the FCI rarely repeating the same set of principles on multiple items and the FMCE often repeating the same principles.…”

“…"Cart on a Ramp" items (items 8-10) ask students to select the force on a cart as it moves up and down an incline. "Coin Toss -Force" items (items [11][12][13] ask students to select the force on a coin tossed in the air. "Force Graph" items (items [14][15][16][17][18][19][20][21] ask students about the force on a toy car as it moves across a low-friction surface; students select from a number of graphs.…”

“…Many studies have applied quantitative analysis methods to explore the structure of conceptual physics instruments. A substantial number of studies have explored the factor structure of the FCI, generally finding inconsistent or unintelligible results [6,7,11,12,29].…”

“…Recently, substantial efforts have been made to apply quantitative techniques to further understand these instruments including factor analysis [6][7][8], cluster analysis [9], and item response theory [10][11][12][13]. In 2016, Brewe, Bruun, and Bearden [14] introduced a new class of quantitative algorithms to analyze the incorrect answers, network analytic methods [15,16].…”