CONTEXT The objective structured clinical examination (OSCE) is comprised of a series of simulations used to assess the skill of medical practitioners in the diagnosis and treatment of patients. It is often used in high-stakes examinations and therefore it is important to assess its reliability and validity.METHODS The published literature was searched (PsycINFO, PubMed) for OSCE reliability estimates (coefficient alpha and generalisability coefficients) computed either across stations or across items within stations. Coders independently recorded information about each study. A meta-analysis of the available literature was computed and sources of systematic variance in estimates were examined.RESULTS A total of 188 alpha values from 39 studies were coded. The overall (summary) alpha across stations was 0.66 (95% confidence interval [CI] 0.62-0.70); the overall alpha within stations across items was 0.78 (95% CI 0.73-0.82). Better than average reliability was associated with a greater number of stations and a higher number of examiners per station. Interpersonal skills were evaluated less reliably across stations and more reliably within stations compared with clinical skills. An advantage of the OSCE over paper-and-pencil tests of knowledge is that the simulations involve more realistic context, content and procedures. For example, in the OSCE, rather than writing an essay about diagnosis, the examinee will encounter an SP and generate a diagnosis based on the clinical interview and examination. An advantage of the OSCE over assessments that use real patients is that the patients are standardised across examinees and thus the patient problems are essentially equivalent and examinee responses and scores are comparable. CONCLUSIONSDespite the apparent advantages of the OSCE over available alternative assessments, the quality of assessment is not guaranteed simply by assembling some standardised problems. The reliability of the assessment is crucial, particularly when the aim of the OSCE is to provide data for high-stakes decisions, as is often the case in medical school assessments. Ideally, the particular problems chosen for the OSCE should not be terribly influential and examinees who pass or fail a given examination should be expected to also pass or fail an alternative examination, should one be given. In other words, the problems in a given examination are intended to tap a portfolio of skills that should be mastered by the medical practitioner. The reliability of the overall examination represents an estimation of the correlation of scores on the given examination with scores on a hypothetical examination composed of the entire portfolio of problems.The research questions driving the current study were:1 What reliability should we expect on average when we develop an OSCE? 2 What is the likely range of such values? 3 What factors appear to influence the expected reliability?The current paper addresses these questions through a quantitative review, which includes descriptions of the effects of the number of...