A systematic review of the reliability of objective structured clinical examination scores

Abstract

CONTEXT The objective structured clinical examination (OSCE) is comprised of a series of simulations used to assess the skill of medical practitioners in the diagnosis and treatment of patients. It is often used in high-stakes examinations and therefore it is important to assess its reliability and validity.METHODS The published literature was searched (PsycINFO, PubMed) for OSCE reliability estimates (coefficient alpha and generalisability coefficients) computed either across stations or across items within stations. Coders independently recorded information about each study. A meta-analysis of the available literature was computed and sources of systematic variance in estimates were examined.RESULTS A total of 188 alpha values from 39 studies were coded. The overall (summary) alpha across stations was 0.66 (95% confidence interval [CI] 0.62-0.70); the overall alpha within stations across items was 0.78 (95% CI 0.73-0.82). Better than average reliability was associated with a greater number of stations and a higher number of examiners per station. Interpersonal skills were evaluated less reliably across stations and more reliably within stations compared with clinical skills. An advantage of the OSCE over paper-and-pencil tests of knowledge is that the simulations involve more realistic context, content and procedures. For example, in the OSCE, rather than writing an essay about diagnosis, the examinee will encounter an SP and generate a diagnosis based on the clinical interview and examination. An advantage of the OSCE over assessments that use real patients is that the patients are standardised across examinees and thus the patient problems are essentially equivalent and examinee responses and scores are comparable. CONCLUSIONSDespite the apparent advantages of the OSCE over available alternative assessments, the quality of assessment is not guaranteed simply by assembling some standardised problems. The reliability of the assessment is crucial, particularly when the aim of the OSCE is to provide data for high-stakes decisions, as is often the case in medical school assessments. Ideally, the particular problems chosen for the OSCE should not be terribly influential and examinees who pass or fail a given examination should be expected to also pass or fail an alternative examination, should one be given. In other words, the problems in a given examination are intended to tap a portfolio of skills that should be mastered by the medical practitioner. The reliability of the overall examination represents an estimation of the correlation of scores on the given examination with scores on a hypothetical examination composed of the entire portfolio of problems.The research questions driving the current study were:1 What reliability should we expect on average when we develop an OSCE? 2 What is the likely range of such values? 3 What factors appear to influence the expected reliability?The current paper addresses these questions through a quantitative review, which includes descriptions of the effects of the number of...

Does Changing Examiner Stations During UK Postgraduate Surgery Objective Structured Clinical Examinations Influence Examination Reliability and Candidates’ Scores?

“…Although these leniency and stringency variations might be a potential issue in communication-related stations, 4,5,17,18 which are generally not dependent on fact (compared with a discipline such as anatomy for which an answer is either correct or not), we did not find a statistical difference in absolute scores between examiners and the content area being examined. In some undergraduate and postgraduate examinations, the use of 2 examiners per scenario is encouraged to minimize inter-examiner variation, though many examinations now rely on just 1 examiner for each OSCE station.…”

“…17 In addition to the reasons discussed earlier, recent work using a NASA task load index rating, identified that excessive workloads occur in OSCE examiners in a similar way that is seen in other professions such as airline pilots. 20 Further research is needed in this relatively unexplored area and other ways to minimize examiner variables need to be considered.…”

Inter-Professional Team Objective Structured Clinical Examination (ITOSCE): Teaching and Assessment Strategies of the Inter Professional Approach

“…Increasing the number of stations, and rater could be effective for improving the reliability of objective structure exams. 53 The other studies showed that with transparent objectives, developed standardized assessment checklist and proper rater training could be improved the ITOSCE reliability.…”

Correlation between Assignments for Professionalism of the Post-graduate Clinical Competency Evaluation and the Pre-graduate Objective Structured Clinical Achievement Test in Dental Education

“…This is based on the global standards proposed by the World Federation for Medical Education (1,2). Cross-sectional studies for each period have been reported regarding education at the pre-graduate clinical practical training and post-graduate clinical training levels (3)(4)(5)(6)(7)(8)(9). However, no studies on a clinical competency examination as an overall evaluation at the end of pregraduate clinical practical training and at the end of OSCAT as a pilot study with the objective of establishing a clinical competency evaluation for professionalism in dental education.…”