Minimum Accepted Competency Examination: Test Item Analysis

Abstract

BackgroundThe minimum accepted competency (MAC) examination in paediatrics was a set of MCQ’s designed to test undergraduate medical students on the most basic, ‘must know’ knowledge as determined by non-faculty paediatric clinicians. The faculty standard set the examination at a relatively difficult 41.2% with only two-thirds of students attaining this score (even though 96% of the same students passed their university paediatric examination). We will now describe the psychometric properties of the examination in a bid to explain these results and provide specific suggestions for future practiceMethodsThe methodology for designing the content, standard setting and administration of the MAC examination has already been described. The examination was set by three groups of undergraduate medical students and paediatric trainee doctors. Test item analysis was performed using difficulty index, discrimination index and Cronbach’s alpha.ResultsTest item difficulty between each group was positively correlated. Correlation of item difficulty with the standard set for each item showed a statistically significant positive relationship. However, for 10 of the items, the mean score did not even reach two standard deviations below the standard set by the faculty. Medical students outperformed the trainee doctors on three items. 18 of 30 items achieved a discrimination index >0.2. Cronbach’s alpha ranged from 0.22-0.59.ConclusionQuestions on respiratory paediatrics and seizures were answered poorly. Despite these topics being major topics within the curriculum, it is possible that the clinical emphasis in these non-faculty derived questions was juxtaposed with the factual recall often required for university examinations. Interestingly, for the questions on which the students performed better than the doctors, the questions were quite specific, and the answer found directly in the curriculum. This reflects how students learn with limited clinical experience and is a good example of assessment being a key driver of learning.The concept of a MAC examination to test basic medical knowledge is feasible and the study presented is an encouraging first step towards this method of assessment. This method is readily transferable to other medical (and allied health) specialties.