Using Item Analysis to Improve Tests

Abstract

Every major textbook in measurement discusses the improvement of a test through the use of item statistics. Most students taking a course in test construction remember that item analysis can be useful in item selection decisions. Unfortunately, they tend to forget that item analyses can also be useful for item revision. Too frequently a person doing an item analysis of a multiple-choice test fails to go beyond computing the item difficulty and discrimination indices. When only this superficial analysis is completed, reasons underlying any item failures can not be discerned and item revision is difficult. It is when the responses to each of the foils have been examined that item revision can be accomplished most effectively.The purpose of this paper is to study the value of using a complete item analysis in rewriting items that have been shown to lack appropriate discrimination power. The researchers were interested in seeing whether it was more efficient to rewrite "poor" items than to write new items to improve the discrimination power of a test. METHODAbout 600-700 students take an introductory educational psychology course at Michigan State University each term. For many years, a part of the evaluation procedure has been multiple-choice examinations. The tests are revised each quarter. Past revision have been conducted primarily by taking previous exam questions that have discriminated well and are still of appropriate content. These were used together with new items written by the instructors. Seldom were individual items rewritten through revision of the foils and even when that was done, no systematic follow-up of these rewritten items was undertaken to determine whether their discriminating powers improved.An instructor familiar with the course content was asked to review a 60-item test given the previous quarter, and select items that were appropriate in content for the test being prepared. A complete item analysis had been conducted on this test showing the difficulty and discrimination indices, as well as the Percent of people in the upper and lower 27% who responded to each alternative? Thirty-two items in this test were found to be still appropriate in content. These 32 items were then examined and 18 were found to be acceptable for use without revision, 14 being in need of revision. These 14 items were revised by an instructor. The major revisions were accomplished by looking at the item analysis data on the foils and revising those foils which were not pulling in the proper direction, that is, those which were not more attractive to the lower group than the upper group. The total time needed 1The discrimination index used was the "D" index. See Findley (1956) and Englehart (1965) for a more thorough discussion. 65 66 JOURNAL OF EDUCATIONAL MEASUREMENT VOL, 4, NO. 2 for revision was about 1 hour. Examples of the type of revision that was done and the consequences are shown in the following items.

The Use of Task-Evoked Pupillary Response as an Objective Measure of Cognitive Load in Novices and Trained Physicians

“…To assess the difficulty and item discrimination, we completed a test item analysis on the MCQs. 23,24 Next, we determined item difficulty by dividing the number of participants answering the question correctly by the total number of participants. By subtracting the novice item difficulty from the expert item difficulty, we determined the item discrimination index.…”

Untitled

“…Besides, learning becomes more interactive as lecturers are able to present feedbacks to students which could facilitate students to improve on those areas. Thus, in the item analysis, it has item revision [11] allows to identify items too difficult or too easy to differentiate between students who have learned the content of the subject and those who have not. The focus here is not just on the lecturers but it involves the students and the items too when it comes to item analysis.…”

“…Items analysis allows us to observe the item characteristics and to improve the quality of the test. In the item analysis, it has item revision [11] allows to identify items too difficult or too easy to differentiate between students who have learned the content of the subject and those who have not. It can also to identify the questions that have distracters not credible.…”

Assessing student progress in academic achievement

“…Furthermore, faculty need to be helped to use item analysis to help revise and improve items, in addition to selecting items. For example, Lange, Lehmann, and Mehrens (1967) found that it took about five times as long to develop a new psychology test item as to revise a n old one, using item analysis, and the revised item was more strongly related to course achievement.…”