Testing the test: Item response curves and test quality

Morris, Gary A.; Branum‐Martin, Lee; Harshman, N. L.; Baker, Stephen; Mazur, Eric; Dutta, Sourish; Mzoughi, Taha; McCauley, Veronica

doi:10.1119/1.2174053

Cited by 79 publications

(103 citation statements)

References 13 publications

Supporting

Mentioning

100

Contrasting

Unclassified

Order By: Relevance

“…For future work, we plan to carry out the qualitative analysis of actual answer choices and patterns as well as quantitative analysis such as item response curves [9,10]. Moreover, we plan to disambiguate our data in order to assess the false-positive ratios by major area and analyze whether they differ among these populations.…”

Section: Discussionmentioning

confidence: 99%

“…Similarly, if we evaluate and unify the systematic error from the validation of the distractor and the context [8][9][10][11], we could derive the systematic error of the FCI, thereby allowing us to compare the results considering these inadequacies without modifying them and to effectively measure the effect of physics education. This conclusion would enable our study to provide a positive and far-reaching effect in the field of physics education research.…”

Section: Discussionmentioning

confidence: 99%

“…Hestenes and colleagues evaluated the validity of the wording and diagrams in its questions [1,7], while Rebello and Zollman analyzed the validity of the distractors in the questions by comparing students' responses to four FCI open-ended questions [8]. Morris and colleagues also evaluated the validity of the distractors by analyzing the item response curves [9,10], and Stewart and colleagues validated the contexts of the questions using a ten-question context-modified test [11].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Validating two questions in the Force Concept Inventory with subquestions

Yasuda

Taniguchi

2013

Phys. Rev. ST Phys. Educ. Res.

View full text Add to dashboard Cite

In this study, we evaluate the structural validity of Q.16 and Q.7 in the Force Concept Inventory (FCI). We address whether respondents who answer Q.16 and Q.7 correctly actually have an understanding of the concepts of physics tested in the questions. To examine respondents' levels of understanding, we use subquestions that test them on concepts believed to be required to answer the actual FCI questions. Our sample size comprises 111 respondents; we derive false-positive ratios for prelearners and postlearners and then statistically test the difference between them. We find a difference at the 0.05 significance level for both Q.16 and Q.7, implying that it is possible for postlearners to answer both questions without an understanding of the concepts of physics tested in the questions; therefore, the structures of Q.16 and Q.7 are invalid. In this study, we only evaluate the validity of these two FCI questions; we do not assess the validity of previous studies that have compared total FCI scores.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Validating two questions in the Force Concept Inventory with subquestions

Yasuda

Taniguchi

2013

Phys. Rev. ST Phys. Educ. Res.

View full text Add to dashboard Cite

show abstract

“…This can be done by graphing the proportion of students who choose the correct response versus the total test score. This is a much easier method that will render useful results for a FASI (Ding & Beichner, 2009;Morris et al, 2006). Full on IRT requires specialized statistical software and large sample sizes from 200 (1 parameter Rausch-arguably not appropriate for a FASI) to 1,000 students (Crocker & Algina, 1986).…”

Section: Carry Out Validation Interviews On Test Questionsmentioning

confidence: 99%

Development and Validation of Instruments to Measure Learning of Expert‐Like Thinking

Adams

Wieman

2010

International Journal of Science Education

368

363

View full text Add to dashboard Cite

This paper describes the process for creating and validating an assessment test that measures the effectiveness of instruction by probing how well that instruction causes students in a class to think like experts about specific areas of science. The design principles and process are laid out and it is shown how these align with professional standards that have been established for educational and psychological testing and the elements of assessment called for in a recent National Research Council study on assessment. The importance of student interviews for creating and validating the test is emphasized, and the appropriate interview procedures are presented. The relevance and use of standard psychometric statistical tests are discussed. Additionally, techniques for effective test administration are presented. Keywords: Assessment; Formative assessment; University; Science education; Evaluation; Assessment designIn recent years, there has been a growing effort to develop assessment tools that target students' development of expert-like mastery of specific science topics. These involve questions that accurately probe whether students understand and apply particular concepts in the manner of a scientist in the discipline. Such assessment tools are intended to measure student learning in courses to provide Formative Assessment of Instruction (FASI). We present the methodology involved in developing and validating such assessment tools. This same methodology works equally well

show abstract

“…Apart from the difficulty of the questions and the effectiveness of distractors (wrong possible choices of multiple choice questions, shown for example in table 3 for question Q2) the item response curve (IRC) analysis can be done on the given questions [16,17]. The results of the IRC give a deeper insight into the quality of the questions, as well as into the abilities of the participating students.…”

Section: Resultsmentioning

confidence: 99%

Assessing learning outcomes from experiments in a science competition

Rovšek

2017

Eur. J. Phys.

View full text Add to dashboard Cite

Teaching quantum physics by the sum over paths approach and GeoGebra simulations AbstractThe Slovene Science competition for primary school students aged between 6 and 12 years will be explained briefly. The competition is based on experiments, which should be done well in advance either at school or at home. An example of the proposed experiment for 11 and 12 year old students from the past year-a syphon-will be presented as well as the assessment questions referring to this particular experiment, which were posed at the competition. A fair proportion of the population took part in the competition. We will present a brief analysis of the obtained data, which offers an insight into the difficulty level of the questions.

show abstract

Testing the test: Item response curves and test quality

Cited by 79 publications

References 13 publications

Validating two questions in the Force Concept Inventory with subquestions

Validating two questions in the Force Concept Inventory with subquestions

Development and Validation of Instruments to Measure Learning of Expert‐Like Thinking

Assessing learning outcomes from experiments in a science competition

Contact Info

Product

Resources

About