Ensuring reliability in UK written tests of general practice: the MRCGP examination 1998–2003

Abstract

Reliability in written examinations is taken very seriously by examination boards and candidates alike. Within general education many factors influence reliability including variations between markers, within markers, within candidates and within teachers. Mechanisms designed to overcome, or at least minimize, the impact of such variables are detailed. Methods of establishing reliability are also explored in the context of a range of assessment situations. In written tests of general practice within the Membership of the Royal College of General Practitioner (MRCGP) examination considerable effort has put been put into achieving acceptable levels of reliability. Current mechanisms designed to ensure high reliability are described and related to the evolution of the written component of the examination. In addition to description of marker selection and training, question development including construct a detailed example of specific and generic marking schedules is provided. Examination results for the Written Paper of the MRCGP from 1998 to 2003 are reported including Cronbach's alpha coefficients and standard error of measurements, mean scores (and SD) and pass rates. In addition individual discrimination scores for each question in the October 2002 paper are shown. Consistent high reliability of the written component of the MRCGP examination provides valuable lessons in terms of selection, training and monitoring of markers as well as practical methods of moderating factors affecting candidate variability. The challenge for examination developers is to carry these important lessons forward into a modernized assessment structure of UK general practice.

Comprehensive and qualifying examinations: A qualitative review of APA-accredited doctoral psychology programs.

“…Our literature review found only one study that discussed and evaluated interrater results of essay-style qualification exams and found reliabilities ranging from 0.59 to 0.69 that utilized a 1–5-point rating with four faculty readers reviewing 26 student responses (Burck & Peterson, 1983). Articles by Forster and Masters (1996), Marsh and Ireland (1987), Munro et al (2005), and Wakeford and Roberts (1984) provide some discussion of processes to assess and improve interrater reliabilities of written essay-style exams.…”

Validation and evaluation of new assessments for the otolaryngology undergraduate medical clerkship

“…These examinations have introduced the elements of both continuous assessment and assessment of ENT practical skills to enhance our undergraduate ENT course. ‘Open form’ or ‘context-rich’ assessments are most suitable for evaluating the application of knowledge and higher level abilities 27 , 28 . Multiple levels of Miller's hierarchical framework of assessment 4 are now included in our course.…”

“…Finally, performance and practical skills are evaluated using an objective structured clinical examination. While it is established that essay and short answer question papers are resource intensive in comparison to machine-markable assessments, 27 the advantage of a dedicated ENT tutor at our institution facilitates the use of these written assessment methods. To save on resources, other institutions may prefer to develop tests of knowledge based on multiple-choice questions.…”

Learning to mark: a qualitative study of the experiences and concerns of medical markers

“…Peer review is an established method for giving feedback and non-threatening quality control in teaching and assessment in a variety of situations [ 14 - 17 ] but its use in review of marking is limited so far. Peer calibration prior to marking a written paper that does not involve meeting and discussing marking interpretation has been described [ 18 , 19 ], but we would advocate face to face structured peer review following the initial marking of a small sample of papers. We have explored this technique [ 13 ], and found markers thought it valuable regardless of their level of experience.…”