A Study of the Reliability of the Clinical Oral Examination in Psychiatry

Abstract

A study of the variability between raters in scoring an oral clinical examination in psychiatry in the format of the certification by the Royal College of Physicians and Surgeons of Canada is reported. A video-taped examination from another centre was rated independently by academic psychiatrists, nonacademic psychiatrists and residents. Considerable inter-rater differences between and within these groups were found. In particular, the averaging of the marks of pairs of raters as occurs in the actual certification resulted in the outcome depending to a considerable degree on the chance pairings of raters. These findings support a number of previous studies and emphasize the need to train examiners and to develop clearer rating criteria.

Objective Structured Clinical Examination: The Assessment of Choice

“…8 It has also been demonstrated that the luck of the draw in selection of examiner and patient played a significant role in the outcome of postgraduate examinations in psychiatry using the traditional method. 6 Published findings of researchers on OSCE from its inception in 1975 to 2004 has reported it to be reliable, valid and objective with cost as its only major drawback. 9 The OSCE however, covers broader range like problem solving, communication skills, decision-making and patient management abilities.…”

“…It is comprised of several "stations" in which examinees are expected to perform a variety of clinical tasks within a specified time period against criteria formulated to the clinical skill, thus demonstrating competency of skills and/ or attitudes. The OSCE has been used to evaluate those areas most critical to performance of health care professionals, such as the ability to obtain/interpret data, problem-solve, teach, communicate, and handle unpredictable patient behavior, [3][4][5][6][7] which are otherwise impossible in the traditional clinical examination. Any attempt to evaluate these critical areas in the old-fashioned clinical case examination will seem to be assessing theory rather than simulating practical performance.…”

“…2,3,4 In the United Kingdom, United States, Canada and indeed most reputable colleges of medicine the OSCE is the standard mode of assessment of competency, clinical skills, and counselling sessions satisfactorily complementing cognitive knowledge testing in essay writing and objective examination. 3,4,5,6 The OSCE is a versatile multipurpose evaluative tool that can be utilized to evaluate health care professionals in a clinical setting. It assesses competency, based on objective testing through direct observation.…”

Simulated patients and objective structured clinical examinations: review of their use in medical education

“…Data gathered by the National Board of Medical Examinations in the USA (1960)(1961)(1962)(1963), involving over 10 000 medical students, showed that the correlation of independent evaluations by two examiners was less than 0.25 (Hubbard et al, 1963). It has also been demonstrated that the luck of the draw in selection of examiner and patient played a significant role in the outcome of postgraduate oral examinations in psychiatry (Leichner et al, 1984). Indeed, Leichner et al went on to suggest that the use of OSCEs may be a feasible means of improving the validity and reliability of oral examinations (Leichner et al, 1986).…”

“…It has also been demonstrated that the luck of the draw in selection of examiner and patient played a significant role in the outcome of postgraduate oral examinations in psychiatry (Leichner et al, 1984). Indeed, Leichner et al went on to suggest that the use of OSCEs may be a feasible means of improving the validity and reliability of oral examinations (Leichner et al, 1986). …”