This case study arises from the need to make more robust and effective quality assurance procedures of the products by automating the final inspection process. The case study explains how the automation of the inspection process was performed in a company from the automotive sector. Knowledge, involvement and commitment of operators and respective managers should not be neglected because their reaction against the change procedures influence the success of any automation performed. The successful introduction of automation contributed to a more efficient process and from the pilot station to the remaining stations problem solving and continuous improvement was evidenced.

BackgroundThe assessment of clinical competences during a national accreditation is a high-stakes process, and the pass mark needs to be settled through robust methods [1]. Since 2012, the MACR certifies trainees in Rheumatology (TR) using both written (multiple-choice questions –MCQ- test) and clinical (OSCE) examinations [2]. For 2013 and 2014 accreditations, OSCE pass mark was set by absolute criterion referencing (APM), while a global rating of borderline performance (BPM) was added for the 2015 process.ObjectivesTo compare the performance of TR during an OSCE according to APM and BPM, and to examine whether correlations between MCQ and OSCE evaluations are affected by OSCE pass mark selection.MethodsForty-three TR presented a MCQ test and an OSCE circuit including 15 stations. Each station (8 minutes long) was assigned to an evaluator who rated candidate performance as “fail”, “borderline” or “above range” using a Likert scale and additionally filled a tailored-checklist. Mean checklist scores (maximum score of 10) from those TR rated as borderline was defined as the BPM for each station. Average of all BPM defined the overall OSCE BPM. A composite OSCE score was obtained for each candidate. Descriptive statistics was used and correlations were examined using Pearson's test and Mann Whitney U test.ResultsFor MCQ test, pass mark was set by criterion as >5.7; OSCE-APM was defined as >6 and composite OSCE-BPM was calculated as >7.4. TR's performance varied depending on the pass mark selected. Mean MCQ test score was 6.6±0.6 with 5 candidates receiving a failing score; mean composite OSCE score was 7.4±0.6 and no candidate received a failing score according to CPM, while 21 (48.8%) received a failing score according to BPM.The BPM score widely varied among stations, ranging from 4.5 to 9.5; candidate's performance also varied across stations and within each station depending on the pass mark; station 2 showed the lowest percentage of TR with APM-passing score (26%) or BPM-passing score (42%); the opposite figure was true regarding station 10 who showed the highest percentage of TR with a passing score (100% and 72%, respectively).MCQ test score correlated with composite OSCE score (r=0.67, P=0.001). Those TR who obtained a passing score according to BPM had higher MCQ scores than those with a failing OSCE score: (median, Q25-Q75) 7.1 (6.4–7.3) vs. 6.3 (5.9–6.7), p=0.001. There were more candidates with a passing OSCE-BPM score distributed in the upper MCQ quartiles than TR with a failing OSCE score (P<0.001).ConclusionsThe way to set a pass mark for an OSCE strongly impacts the performance of TR during a national accreditation process but poorly affects correlations between clinical and written examinations.ReferencesWass V, et al. Assessment of clinical competence. Lancet 2001;357:945–9.Pascual-Ramos V, et al. Performance of an objective structured clinical examination in a national certification process of trainees in rheumatology. Reumatol Clin 2015;11:215–20.Disclosure of InterestNone declared