Differences in Item Statistics Between First Order and Second Order Laboratory Practical Examination Questions

Abstract

Previous research has shown that medical students perform better on first order laboratory practical questions than they do on second order questions. First order questions are defined as requiring the student to only identify a tagged structure. Second order questions ask the student to apply some information about the tagged structure rather than to only identify the structure (i.e. “What is the innervation of the tagged structure?”). This study followed up on the previous research by examining item statistics of first and second order questions to determine individual item quality. Statistics of interest were item difficulty and discrimination index. Item difficulty was defined as the percentage of students getting a particular item correct, so the higher the percentage, the easier the item. Discrimination index (DI) was defined as the difference in percent correct between higher and lower performing students on a particular item. Quality questions typically have a moderate difficulty of approximately 80% correct and a DI of greater than .30. The expectation was that second order questions would continue to show increased difficulty but have a higher DI and potentially be of higher quality than first order questions. The laboratory practical studied was a traditional steeplechase‐style examination where students had a single question at approximately 60 stations. Students had one minute at each station and could not return to any station during or after the examination. During setup, the instructors purposely set about 20% of the items to be second order questions. Responses were analyzed for question difficulty and DI, which was the difference in performance on an item between the top 27% of the class and the bottom 27%. One hundred and seventy two students took the 57‐question examination. There were 13 second order questions (22.8%). The examination average score was 75.04 ± 12.6% and the average DI was .293 ± .14. First order questions averaged 76.18% correct with a DI of .31 while second order questions averaged 72.81% and had a DI of .25. Only one (7.7%) of the second order questions had a difficulty between 75–85% and a DI above .30 while seven (15.9%) of the first order questions achieved these statistics. A post hoc analysis of the 596 total errors made on the thirteen second order questions was also conducted. Examining each of the wrong responses revealed that approximately ten percent of the errors were due to the students giving the first order identification rather than the second order information that was being sought. Initial results indicate that second order questions did not perform better than first order questions in a laboratory practical setting based on traditional item statistics. Instructors need to be aware that these types of assessment questions may not be good indicators of a student’s foundational knowledge as well as their ability to apply that knowledge beyond straight forward identification of anatomical structures. Further investigation should be done to see if these statist...