Tools for the direct observation and assessment of psychomotor skills in medical trainees: a systematic review

Abstract

Context The Accreditation Council for Graduate Medical Education (ACGME) Milestone Project mandates programmes to assess the attainment of training outcomes, including the psychomotor (surgical or procedural) skills of medical trainees. The objectives of this study were to determine which tools exist to directly assess psychomotor skills in medical trainees on live patients and to identify the data indicating their psychometric and edumetric properties. Methods An electronic search was conducted for papers published from January 1948 to May 2011 using the PubMed, Education Resource Information Center (ERIC), Cumulative Index to Nursing and Allied Health Literature (CINAHL) and Web of Science electronic databases and the review of references in article bibliographies. A study was included if it described a tool or instrument designed for the direct observation of psychomotor skills in patient care settings by supervisors. Studies were excluded if they referred to tools that assessed only clinical or non‐technical skills, involved non‐medical health professionals, or assessed skills performed on a simulator. Overall, 4114 citations were screened, 168 (4.1%) articles were reviewed for eligibility and 51 (1.2%) manuscripts were identified as meeting the study inclusion criteria. Three authors abstracted and reviewed studies using a standardised form for the presence of key psychometric and edumetric elements as per ACGME and American Psychological Association (APA) recommendations, and also assigned an overall grade based on the ACGME Committee on Educational Outcome Assessment grading system. Results A total of 30 tools were identified. Construct validity based on associations between scores and training level was identified in 24 tools, internal consistency in 14, test–retest reliability in five and inter‐rater reliability in 20. The modification of attitudes, knowledge or skills was reported using five tools. The seven‐item Global Rating Scale and the Procedure‐Based Assessment received an overall Class 1 ACGME grade and are recommended based on Level A ACGME evidence. Conclusions Numerous tools are available for the assessment of psychomotor skills in medical trainees, but evidence supporting their psychometric and edumetric properties is limited.

A systematic review of assessments for procedural skills in physiotherapy education / Assessment von prozeduralen Fähigkeiten in der physiotherapeutischen Ausbildung: Ein systematischer Review

“…In contrast, a considerably greater number of assessments are available for use in medical education: McKinley et al (2008) included 85 different scales in their review of assessments used in medical education. Our findings were similar to those of Jelovsek et al (2013), who found that there was limited reporting of measurement properties. Bould et al (2009) suggested that procedure unspecific assessments tended to miss errors in safety issues.…”

“…Four recent systematic reviews were identified that reported the assessment of procedural skills in health professions education (Bould, Crabtree, & Naik, 2009;Jelovsek et al, 2013;McKinley et al, 2008;Morris et al, 2012). In general, these reviews focussed on medical education and few assessments relevant for use by allied health professions were identified.…”

“…Therefore, following the initial teaching of procedural skills, physiotherapy educators need valid and reliable assessment tools to evaluate whether procedural competency of students is sufficient for practice. Assessment of procedural skills has been extensively researched in surgical education (Jelovsek, Kow, & Diwadkar, 2013). Some assessments exist, which can be used for procedures in nursing education (Morris, Gallagher, & Ridgway, 2012).…”

Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence

“…OSATS ratings in the operating room demonstrated consistent expert-novice discrimination, but other aspects of the validity argument were inconsistently evaluated. However, a more recent systematic review of assessment tools for direct observation of psychomotor skills in medical learners recommended the OSATS global rating scale be implemented as a core component of a training program's evaluation methods (Class 1 ACGME grade) (Jelovsek et al 2013). This recommendation was based on assessing the global rating scale validity, reliability and utility in 12 studies.…”

“…While previous reviews on technical skills assessment have either focussed on an overview of all of these tools (Moorthy 2003;Cook et al 2013;van Hove et al 2010), or examined one specific technical skill such as laparoscopic surgery (Aggarwal et al 2004), none have provided an indepth examination of the validity evidence supporting the intended use of a specific tool, and in particular we found no reviews focused on the OSATS. As the OSATS is the most widely-used technical skills assessment tool for both simulation-based (Cook et al 2013), and non-simulation based (van Hove et al 2010) technical skills assessment, and as the OSATS global rating scale is recommended for the direct assessment of psychomotor skills in medical learners (Jelovsek et al 2013), we believe the time is ripe for an in-depth examination of the validity evidence underlying the intended use of this tool.…”