Tools for Direct Observation and Assessment of Clinical Skills of Medical Trainees

Abstract

Context Direct observation of medical trainees with actual patients is important for performance-based clinical skills assessment. Multiple tools for direct observation are available, but their characteristics and outcomes have not been compared systematically.Objectives To identify observation tools used to assess medical trainees' clinical skills with actual patients and to summarize the evidence of their validity and outcomes.Data Sources Electronic literature search of PubMed, ERIC, CINAHL, and Web of Science for English-language articles published between 1965 and March 2009 and review of references from article bibliographies. Study SelectionIncluded studies described a tool designed for direct observation of medical trainees' clinical skills with actual patients by educational supervisors. Tools used only in simulated settings or assessing surgical/procedural skills were excluded. Of 10 672 citations, 199 articles were reviewed and 85 met inclusion criteria. Data ExtractionTwo authors independently abstracted studies using a modified Best Evidence Medical Education coding form to inform judgment of key psychometric characteristics. Differences were reconciled by consensus.Results A total of 55 tools were identified. Twenty-one tools were studied with students and 32 with residents or fellows. Two were used across the educational continuum. Most (n=32) were developed for formative assessment. Rater training was described for 26 tools. Only 11 tools had validity evidence based on internal structure and relationship to other variables. Trainee or observer attitudes about the tool were the most commonly measured outcomes. Self-assessed changes in trainee knowledge, skills, or attitudes (n=9) or objectively measured change in knowledge or skills (n = 5) were infrequently reported. The strongest validity evidence has been established for the Mini Clinical Evaluation Exercise (Mini-CEX). ConclusionAlthough many tools are available for the direct observation of clinical skills, validity evidence and description of educational outcomes are scarce.

Teamwork Assessment in Internal Medicine: A Systematic Review of Validity Evidence and Outcomes

“…26 This framework includes five categories of validity evidence: 1) content (the degree to which the tool content reflects the construct being measured); 2) response process (training of raters to use the tool); 3) internal structure (instrument reliability including internal consistency, interrater, intrarater, and test-retest reliability); 4) relationships to other variables (relationship between scores and other variables measuring the same construct); and 5) consequences (outcomes associated with tool scores). Kirkpatrick's hierarchy was used to categorize outcomes as satisfaction/opinion, knowledge and skills, behaviors, and patient outcomes.…”

Musculoskeletal education in US medical schools: lessons from the past and suggestions for the future

“…Based on Miller's framework (Table 1), assessment of medical students in the preclinical years is usually limited to demonstrating that students have knowledge and competence. In the clinical years, a number of tools have been developed to assess performance and action, such as the mini-Clinical Evaluation Exercise (CEX) [50,51]; however, it is not known how frequently these are used specifically to assess musculoskeletal performance.…”

Designing Assessment Programmes for the Model Curriculum for Emergency Medicine Specialists

“…Although it is beyond the scope of this paper to describe the utility of these methods in detail, a summary of the utility of some the assessment methods described above is presented in the supplementary online appendices (Appendices 8-10). In addition, several excellent reviews currently exist which describe the utility of these assessment methods in general postgraduate training 8,9,[25][26][27] and with reference to the ACGME 21,22,[28][29][30][31][32][33] and CanMEDS framework 10,23 .…”