FRI0606 The Method Used To Set Pass Marks in An Objective Structured Clinical Examination (OSCE) Determines The Overall Performance of Candidates during An Accreditation Process

Abstract

BackgroundThe assessment of clinical competences during a national accreditation is a high-stakes process, and the pass mark needs to be settled through robust methods [1]. Since 2012, the MACR certifies trainees in Rheumatology (TR) using both written (multiple-choice questions –MCQ- test) and clinical (OSCE) examinations [2]. For 2013 and 2014 accreditations, OSCE pass mark was set by absolute criterion referencing (APM), while a global rating of borderline performance (BPM) was added for the 2015 process.ObjectivesTo compare the performance of TR during an OSCE according to APM and BPM, and to examine whether correlations between MCQ and OSCE evaluations are affected by OSCE pass mark selection.MethodsForty-three TR presented a MCQ test and an OSCE circuit including 15 stations. Each station (8 minutes long) was assigned to an evaluator who rated candidate performance as “fail”, “borderline” or “above range” using a Likert scale and additionally filled a tailored-checklist. Mean checklist scores (maximum score of 10) from those TR rated as borderline was defined as the BPM for each station. Average of all BPM defined the overall OSCE BPM. A composite OSCE score was obtained for each candidate. Descriptive statistics was used and correlations were examined using Pearson's test and Mann Whitney U test.ResultsFor MCQ test, pass mark was set by criterion as >5.7; OSCE-APM was defined as >6 and composite OSCE-BPM was calculated as >7.4. TR's performance varied depending on the pass mark selected. Mean MCQ test score was 6.6±0.6 with 5 candidates receiving a failing score; mean composite OSCE score was 7.4±0.6 and no candidate received a failing score according to CPM, while 21 (48.8%) received a failing score according to BPM.The BPM score widely varied among stations, ranging from 4.5 to 9.5; candidate's performance also varied across stations and within each station depending on the pass mark; station 2 showed the lowest percentage of TR with APM-passing score (26%) or BPM-passing score (42%); the opposite figure was true regarding station 10 who showed the highest percentage of TR with a passing score (100% and 72%, respectively).MCQ test score correlated with composite OSCE score (r=0.67, P=0.001). Those TR who obtained a passing score according to BPM had higher MCQ scores than those with a failing OSCE score: (median, Q25-Q75) 7.1 (6.4–7.3) vs. 6.3 (5.9–6.7), p=0.001. There were more candidates with a passing OSCE-BPM score distributed in the upper MCQ quartiles than TR with a failing OSCE score (P<0.001).ConclusionsThe way to set a pass mark for an OSCE strongly impacts the performance of TR during a national accreditation process but poorly affects correlations between clinical and written examinations.ReferencesWass V, et al. Assessment of clinical competence. Lancet 2001;357:945–9.Pascual-Ramos V, et al. Performance of an objective structured clinical examination in a national certification process of trainees in rheumatology. Reumatol Clin 2015;11:215–20.Disclosure of InterestNone declared