Analyzing force concept inventory with item response theory

Abstract

PACS number(s): 01.40.Fk AbstractItem Response Theory (IRT) is a popular assessment method used in education measurement, which builds on an assumption of a probability framework connecting students' innate ability and their actual performances on test items. The model transforms students' raw test scores through a nonlinear regression process into a scaled proficiency rating, which can be used to compare results obtained with different test questions. IRT also provides a theoretical approach to address ceiling effect and guessing. We applied IRT to analyze the Force Concept Inventory (FCI). The data was collected from 2802 students taking intro level mechanics courses at The Ohio State University. The data was analyzed with a 3-parameter item response model for multiple choice questions. We describe the procedures of the analysis and discuss the results and the interpretations. The analysis outcomes are compiled to provide a detailed IRT measurement metric of the FCI, which can be easily referenced and used by teachers and researchers for a range of assessment applications.

Dividing the Force Concept Inventory into two equivalent half-length tests

“…Analysis of the data set used in this paper suggests that student scores on the two half-length tests reasonably follow a normal distribution (R 2 > 0.97) and that the condition of this data set is appropriate for conducting IRT analysis. Similar results have also been reported in our previous study on IRT application to FCI data [18].…”

“…The typical set of assessment parameters includes test discrimination, test difficulty, and the guessing factor. Existing research has shown that IRT can be used to evaluate features of the FCI test [18,19]. The advantages of using an IRT-based method is that it can maintain assessment consistency when student populations have very different mean scores and that the estimated assessment parameters can help extend the assessment scale into questions not used on the test [16].…”

Gender fairness within the Force Concept Inventory

“…Many different IRT models have been applied to the FCI [4,[47][48][49]. Of these studies, only Wang and Bao [4] reported the item characteristic curves which show how well the data fit the IRT model; none of their curves showed the dramatic departures from fit reported for some of the engineering conceptual inventories examined by Jorion et al [12], indicating that the items in the FCI are generally performing properly. Only Popp et al [47] reported results disaggregated by gender; these results are describe in Sec.…”

“…Wang and Bao calculated CTT difficulty and discrimination parameters for the FCI pretest of 2800 students at a large university in the U.S. [4]. Five of the items had difficulty parameters outside of the desired range (items 1, 6, and 12 with P > 0.8 and items 17 and 26 with P < 0.2), with none having discrimination less than 0.2.…”

“…The Force Concept Inventory (FCI) [1] has been studied using tools such as factor analysis [2,3], item response theory [4,5], and network analysis [6]. Though these investigations have probed the structure and validity of the test, they have primarily treated student data as a single undifferentiated sample and have not studied gender effects.…”