Post-examination interpretation of objective test data: Monitoring and improving the quality of high-stakes examinations: AMEE Guide No. 66

Abstract

The purpose of this Guide is to provide both logical and empirical evidence for medical teachers to improve their objective tests by appropriate interpretation of post-examination analysis. This requires a description and explanation of some basic statistical and psychometric concepts derived from both Classical Test Theory (CTT) and Item Response Theory (IRT) such as: descriptive statistics, explanatory and confirmatory factor analysis, Generalisability Theory and Rasch modelling. CTT is concerned with the overall reliability of a test whereas IRT can be used to identify the behaviour of individual test items and how they interact with individual student abilities. We have provided the reader with practical examples clarifying the use of these frameworks in test development and for research purposes.

Psychometric evaluation of a knowledge based examination using Rasch analysis: An illustrative guide: AMEE Guide No. 72

“…This function is depicted graphically in an ICC. We have already shown how to draw the ICC elsewhere (Tavakol & Dennick 2012a, 2012b. We indicated that if the student ability is equal to the item difficulty (B-D ¼ 0), the probability that the student will answer the question correctly is 50% which can be seen in the ICC.…”

“…For example, in a test we would not expect to see low performing students having a high probability of answering a given item correctly. We have explained how to draw an ICC using student ability and item ability elsewhere (Tavakol & Dennick 2012a, 2012b …”

“…Medical educators and standard setters need to provide a stable and predictable measure of student performance over time to minimise sources of variation in examination data. The postexamination analysis of objective test data can provide the diagnostic feedback to not only improve the validity and reliability of assessments, but also improve curricula and teaching strategies (Tavakol & Dennick 2011b, 2012a, 2012b.…”

The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part II: Organisation & Administration

“…Further information on the impact of scoring rubrics can be found in AMEE Guides 49, 54 and 66 (Pell et al 2010;Tavakol & Dennick 2011;Tavakol & Dennick 2012).…”

Tests, Quizzes, and Self-Assessments: How to Construct a High-Quality Examination

“…Table 2 lists frequently used parameters for quantitative evaluation for each level. Statistical details are available in a comprehensive guide from the Association for Medical Education in Europe [32].…”