Resident and Faculty Attitudes Toward the Dutch Radiology Progress Test as It Transitions from a Formative to a Summative Measure of Licensure Eligibility

Abstract

Background Progress testing, a regularly administered comprehensive test of a complete knowledge domain, usually serves to provide learners feedback and has a formative nature. Objective Our study aimed to investigate the acceptability of introducing a summative component in the postgraduate Dutch Radiology Progress Test (DRPT) among residents and program directors in a competency-based training program. Methods A 15-item questionnaire with 3 items on acceptability of summative postgraduate knowledge testing, 7 on acceptability of the summative DRPT regulations, 4 on self-reported educational effects, and 1 open comment item was distributed nationally among 349 residents and 81 radiology program directors. Results The questionnaire was filled out by 330 residents (95%) and 48 (59%) program directors. Summative postgraduate knowledge testing was regarded as acceptable by both groups, but more so by program directors than residents. The transition toward summative assessment in the DRPT was received neutrally to slightly positively by residents, while program directors regarded it as an improvement and estimated the summative criteria to be lighter and less stressful than did residents. The residents’ self-reported educational effects of summative assessment in the DRPT were limited, whereas program directors expected a greater end-of-training knowledge improvement than residents. Conclusions Both residents and program directors support summative postgraduate knowledge testing, although it is more accepted by program directors. Residents receive summative radiological progress testing neutrally to slightly positively, while program directors generally value it more positively than residents. Directors should be aware of these different perspectives when introducing or developing summative progress testing in residency programs.

Introducing Summative Progress Testing in Radiology Residency: Little Change in Residents’ Test Results After Transitioning from Formative Progress Testing

“…The DRPT's summative pass/fail criterion was defined as follows [19]: residents must obtain a pass score for at least 3 individual tests that are taken in PGYs 2.5-5, while tests in PGYs 0-2.5 only serve formative purposes. Final year residents who are deemed to be at risk-based on their test results in PGYs during which newly enrolling residents were given the opportunity to choose (at the latest by June 2015) which of the 2 residency program variants they wanted to follow until graduation: either the former curriculum with an exclusively formative DRPT or the revised program with DRPT pass requirement 2.5-4-of failing to reach 3 pass scores by the end of residency are obliged to take (and pass) the examination for the European Diploma in Radiology (EDiR) of the European Society of Radiology before completion of residency.…”

“…Medical educators have often used progress testing as a purely formative tool [10,12]. In recent years, however, a transition to summative progress testing has taken place in several institutions [17][18][19]. An important reason for this transitioning may be the need for accountability that weighs upon medical educators, as the aim of CBME is to demonstrate that graduating trainees meet professional standards [20].…”

“…Following this, transitioning to summative progress testing may be expected to increase the knowledge level in participants. This presumed positive educational effect has been an additional reason for some program directors to transition from formative to summative progress testing in resident training [ 19 ]. In case transitioning does not induce a positive effect, then at least it should not impair the trainee’s knowledge level.…”

“…In July 2014, this program was thoroughly revised by introducing radiological entrustable professional activities (EPAs) and by merging the training programs of radiology and nuclear medicine—which used to be separate residency programs—into 1 combined national radiology residency program. Moreover, this revision comprised transitioning the DRPT to include a summative pass requirement (defined below) with the aim to stimulate learning and to meet the need for accountability [ 19 , 20 ]. As part of this transition, residents who started radiology training between July 2014 and July 2015 were given the opportunity to choose, at the latest by June 2015, which of the 2 program variants they wanted to follow until graduation: either the former curriculum with an exclusively formative DRPT or the revised program with DRPT pass requirement (see Fig.…”

“…The DRPT’s summative pass/fail criterion was defined as follows [ 19 ]: residents must obtain a pass score for at least 3 individual tests that are taken in PGYs 2.5–5, while tests in PGYs 0–2.5 only serve formative purposes. Final year residents who are deemed to be at risk—based on their test results in PGYs 2.5–4—of failing to reach 3 pass scores by the end of residency are obliged to take (and pass) the examination for the European Diploma in Radiology (EDiR) of the European Society of Radiology before completion of residency.…”