Anatomy of a physics test: Validation of the physics items on the Texas Assessment of Knowledge and Skills

Marshall, Jill; Hagedorn, Eric; O’Connor, J. Michael

doi:10.1103/physrevstper.5.010104

Cited by 8 publications

(10 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As for the distractors, Answer Choice 1 (only gravity acts) is attractive to students at the very lowest end of the total score range (<2). Answer Choice 3 (gravity and a force in the direction of motion act) preferentially attracts students at the low end of the range as well (2)(3)(4)(5)(6)(7)(8)(9)(10). The IRC for Answer Choice 5 (gravity, a force in the direction of motion, and a force pointing from the center of the circle outward) shows a broad peak in the range of total scores of 9-16, while the IRC for Answer Choice 4 (gravity, a force exerted by the channel toward the center of the circle, and a force in the direction of motion) shows a broad peak in the total score range of 14-23.…”

Section: Additional Fci Insights From Irc Analysismentioning

confidence: 99%

“…Other approaches include the nonparametric approach of Ramsay's TestGraf, a graphical approach to item analysis that makes fewer assumptions than standard IRT. 10 For comparison with the approach of Wang and Bao and as an example to teachers and researchers looking for a practical approach to investigate item functionality, we applied IRC analysis to all 30 questions on the FCI. We used total score as a proxy for ability level, an approach supported by the findings of Wang and Bao-see their Fig.…”

Section: Insights From Recent Analyses Compared With Irc Analysismentioning

confidence: 99%

See 1 more Smart Citation

An item response curves analysis of the Force Concept Inventory

Morris

Harshman

Branum‐Martin

et al. 2012

American Journal of Physics

View full text Add to dashboard Cite

Several years ago, we introduced the idea of item response curves (IRC), a simplistic form of item response theory (IRT), to the physics education research community as a way to examine item performance on diagnostic instruments such as the Force Concept Inventory (FCI). We noted that a full-blown analysis using IRT would be a next logical step, which several authors have since taken. In this paper, we show that our simple approach not only yields similar conclusions in the analysis of the performance of items on the FCI to the more sophisticated and complex IRT analyses but also permits additional insights by characterizing both the correct and incorrect answer choices. Our IRC approach can be applied to a variety of multiple-choice assessments but, as applied to a carefully designed instrument such as the FCI, allows us to probe student understanding as a function of ability level through an examination of each answer choice. We imagine that physics teachers could use IRC analysis to identify prominent misconceptions and tailor their instruction to combat those misconceptions, fulfilling the FCI authors’ original intentions for its use. Furthermore, the IRC analysis can assist test designers to improve their assessments by identifying nonfunctioning distractors that can be replaced with distractors attractive to students at various ability levels.

show abstract

Section: Additional Fci Insights From Irc Analysismentioning

confidence: 99%

Section: Insights From Recent Analyses Compared With Irc Analysismentioning

confidence: 99%

An item response curves analysis of the Force Concept Inventory

Morris

Harshman

Branum‐Martin

et al. 2012

American Journal of Physics

View full text Add to dashboard Cite

show abstract

“…Two common perspectives on test development are Classical Test Theory (CTT) [32] and Item Response Theory (IRT) [33]. The majority of conceptual assessments in physics, at both the introductory and upper-division levels have been validated using CTT, while only a small number have been developed or analyzed using IRT [34][35][36][37]. One significant drawback of CTT is that all test statistics are population dependent.…”

Section: Validationmentioning

confidence: 99%

Development and uses of upper-division conceptual assessments

Wilcox

Caballero

Baily

et al. 2015

Phys. Rev. ST Phys. Educ. Res.

View full text Add to dashboard Cite

[This paper is part of the Focused Collection on Upper Division Physics Courses.] The use of validated conceptual assessments alongside conventional course exams to measure student learning in introductory courses has become standard practice in many physics departments. These assessments provide a more standard measure of certain learning goals, allowing for comparisons of student learning across instructors, semesters, institutions, and pedagogies. Researchers at the University of Colorado Boulder have developed several similar assessments designed to target the more advanced physics of upper-division classical mechanics, electrostatics, quantum mechanics, and electrodynamics courses. Here, we synthesize the existing research on our upper-division assessments and discuss some of the barriers and challenges associated with their development, validation, and implementation as well as some of the strategies we have used to overcome these barriers.

show abstract

“…[4,5]. Practitioners are hearing more about Item Response Theory (IRT) and unidimensionality-the idea that the likelihood of a correct response to any of the items in a set depends on only a single latent ability in the student.…”

Section: Introductionmentioning

confidence: 99%

Multidimensional student skills with collaborative filtering

Bergner¹,

Rayyan²,

Seaton³

et al. 2013

AIP Conference Proceedings

View full text Add to dashboard Cite

Despite the fact that a physics course typically culminates in one final grade for the student, many instructors and researchers believe that there are multiple skills that students acquire to achieve mastery. Assessment validation and data analysis in general may thus benefit from extension to multidimensional ability. This paper introduces an approach for model determination and dimensionality analysis using collaborative filtering (CF), which is related to factor analysis and item response theory (IRT). Model selection is guided by machine learning perspectives, seeking to maximize the accuracy in predicting which students will answer which items correctly. We apply the CF to response data for the Mechanics Baseline Test and combine the results with prior analysis using unidimensional IRT.

show abstract

Anatomy of a physics test: Validation of the physics items on the Texas Assessment of Knowledge and Skills

Cited by 8 publications

References 7 publications

An item response curves analysis of the Force Concept Inventory

An item response curves analysis of the Force Concept Inventory

Development and uses of upper-division conceptual assessments

Multidimensional student skills with collaborative filtering

Contact Info

Product

Resources

About