An argument-based approach to the validation of UHTRUST: can we measure how recent graduates can be trusted with unfamiliar tasks?

Abstract

There is a need for valid methods to assess the readiness for clinical practice of medical graduates. This study evaluates the validity of Utrecht Hamburg Trainee Responsibility for Unfamiliar Situations Test (UHTRUST), an authentic simulation procedure to assess whether medical trainees are ready to be entrusted with unfamiliar clinical tasks near the highest level of Miller's pyramid. This assessment, in which candidates were judged by clinicians, nurses and standardized patients, addresses the question: can this trainee be trusted with unfamiliar clinical tasks? The aim of this paper is to provide a validity argument for this assessment procedure. We collected data from various sources during preparation and administration of a UHTRUST-assessment. In total, 60 candidates (30 from the Netherlands and 30 from Germany) participated. To provide a validity argument for the UHTRUST-assessment, we followed Kane's argument-based approach for validation. All available data were used to design a coherent and plausible argument. Considerable data was collected during the development of the assessment procedure. In addition, a generalizability study was conducted to evaluate the reliability of the scores given by assessors and to determine the proportion of variance accounted by candidates and assessors. It was found that most of Kane's validity assumptions were defendable with accurate and often parallel lines of backing. UHTRUST can be used to compare the readiness for clinical practice of medical graduates. Further exploration of the procedures for entrustment decisions is recommended.

Assessing Competences in Medical Students with a Newly Designed 360- Degree Examination of a Simulated First Day of Residency: A Feasibility Study

“…Even though it has been shown for mini-CEX that rater training had no effect on the reliability of the exam scores [30] it is an unconditional requirement to reach solid reliability for the assessment of competences [11]. Another study also showed that rater training had a positive effect on the quality of inferences made by raters in competence modeling [31].…”

“…Design of the 360-degree competence-based assessment This 360-degree assessment for undergraduate medical students close to their graduation was based on the top 10 competencies identified by Fürstenberg et al [17] and comprised an authentic simulation for medical students in the role of a beginning resident on a busy first day in hospital developed earlier [11]. The assessment includes five phases, which can be completed by 30 participants per day, six groups of five students each (Figure 1).…”

“…After the last conversation with a simulated patient, every participant and every simulated patient completed a Gap-Kalamazoo Communication Skills Assessment Form (GAKCSA) [23] During the patient management phase, participants completed a Post-Encounter Form (modified according to Durning et al [24]) for each simulated patient. Meanwhile the supervisors filled out a Form for Facets of Competence (FOC, modified according to [11]) for their assessment of the participants' management of every patient and documented every telephone encounter with a participant in a logbook. The nurses completed an FOC form for each single disturbance per participant.…”

“…Our aim was to extend the competence-based assessment developed by Wijnen-Meijer et al [11] based on the top 10 competences ranked by German physicians to a 360-degree examination simulating the first day of residency. We wanted to explore whether it was feasible to implement such a type of competence-based examination for a greater cohort of undergraduate medical students from three different medical schools.…”

Entrustability Scales

“…Accordingly, a tool aligned to the construct of competency progression would incorporate an expert's perception of the trainee's increasing clinical ability. 23 Entrustability scales are a species of construct-aligned anchor scales because they align with the expertise and priorities of clinician-educators. The Zwisch scale, 24 a behaviorally anchored ordinal (1-4) scale used to grade the degree of guidance necessary during a technical procedure, is a great simple example.…”