Collaborative Examination Item Review Process in a Team-Taught, Self-Care Sequence

Abstract

Objective. To improve examination item quality by educating and involving course instructors in evidence-based item review and encouraging use of this process in future courses. Methods. A peer-review process was implemented in a 2-course sequence (intervention) that involved training and review sessions before each examination and was compared to the previous year's courses (control). Instructors completed a presurvey and postsurvey on training, experience, self-confidence, and self-rated success in multiple-choice item writing. Statistics were calculated for all items in the control and intervention sequences and compared using independent t tests. Items also were classified into levels based on difficulty and discrimination, and distribution into these levels was compared between sequences with independent t tests. Results. No significant difference was found between control and intervention sequence items with regard to mean difficulty (86.3% and 84.4%) or discrimination (0.23-and 0.25), respectively, although item classification distribution did appear to change between the control and intervention sequences' subjective feelings of confidence, and success in item writing increased between presurvey and postsurvey. Confidence in ability to peer-review test items and to implement a formal item evaluation process also increased. Conclusion. Item statistics did not change significantly, but reviewed and edited items distributed more favorably into item statistic-based categories. This method of review positively affected instructors' perceptions of their item-writing confidence and success and improved self-rated opinions of their ability to edit items and train others to do so.

Differences in Multiple-Choice Questions of Opposite Stem Orientations Based on a Novel Item Quality Measure

“…39 It is therefore reasonable to leverage the potential synergy of item difficulty and discrimination when used contextually to make decisions on the quality of items on an assessment. 19 As evidence of the validity of this novel measure, variations of this approach have been used to evaluate the effect of faculty training on quality of examination items written, 40 item complexity on item quality, 13 impact of automatic item elimination based on item quality, 26 and relationship between distractor efficiency and item quality. 41 However, to the best of our knowledge, this is the first time that this measure has been used to investigate the potential difference in quality between items of opposite stem orientation.…”