Balancing content of computerized adaptive testing for the Force Concept Inventory

Abstract

As a method to shorten the test time of the Force Concept Inventory (FCI), we previously suggested the use of Computerized Adaptive Testing (CAT). CAT is the process of administering a test on a computer, with items (i.e., questions) selected based upon the responses of the examinee to prior items. As a step to develop a CAT-based version of the FCI (FCI-CAT), we previously examined the optimal test length of the FCI-CAT such that accuracy and precision [which were measured in terms of root-mean-square error (RMSE)] of Cohen's d would be comparable to that of the full FCI for a given class size. The objective of this paper is to address an issue in our previous study to improve the FCI-CAT. We consider content balancing ensuring that the same set of concepts assessed in the original test is covered in the CAT administration for each respondent. To balance content in CAT, the percentage of items to be administered from each subgroup is defined in advance. Doing so ensures that items from each subgroup are administered. We conducted a Monte Carlo simulation to analyze how implementing an algorithm of content balancing affects the RMSE of Cohen's d. As a result, we found that, for a class size of 40, the increase of the RMSE due to content balancing is 6%-7% for test lengths of 2-5 items and less than 1% if the test length is larger than 13 items. This result indicates that for a sufficiently large test length (say, larger than 13 items), content balancing does not compromise the accuracy and precision of the FCI-CAT. Hence, we recommend that the FCI-CAT incorporate content balancing, provided the test length is larger than 13 items.

Error‐based simulation as a thought experimental method for changing “motion implies a force” misconception: An evaluation

“…The FCI is used frequently in Japan (where our experiment was conducted) to measure students' understanding of mechanics. For example, Yasuda et al (2011) reported that the average score of 16 undergraduate students who attended an introductory lecture on mechanics was 70% (21 points), and the score of 5 of these students was below 60%. Kuroda et al (2017) reported the average scores of undergraduate students who attended lectures on introductory physics.…”