Measurement of perception and interpretation skills during radiology training: utility of the script concordance approach

Abstract

Imaging specialties require both perceptual and interpretation skills. Except in very simple cases, data perception and interpretation vary among clinicians. This variability makes for difficulty in measuring these skills with traditional assessment tools. The script concordance approach is conceived to allow standardized assessment in contexts of uncertainty. In this exploratory study, the authors tested the usefulness of the approach for assessment of perceptual and interpretation skills in radiology. A perception test (PT) and an interpretation test (IT) were designed according to the approach. Both tests used plain chest X-rays. Three groups were tested: clerkship students (20), junior residents (R1-R3; 20), senior residents (R4-R5; 20). Eleven certified radiologists, all currently appointed to chest reading, provided the answers by aggregate scoring method. Statistics included descriptive, ANOVA, regression analysis, Pearson and Spearman correlation coefficients. Cronbach alpha values were 0.79 and 0.81 for the PT and IT respectively. Score progression was statistically significant in both tests. Perception scores progressed more rapidly than interpretation scores during training. Effect size was large in discriminating low versus higher level of expertise, 2.2 (PT) and 1.6 (IT). The Pearson correlation coefficient between both tests was 0.58. Cronbach alpha coefficient values indicate reasonable reliability for both tests. The linear progression of scores, each at its own pace, and the positive and moderate magnitude of the Pearson correlation coefficient are arguments suggesting measurement of two different skills. More studies are necessary to document the approach usefulness for assessment in radiology training.

Le test de concordance de script comme outil d’évaluation formative en réanimation médicale

“…Ainsi, nous avons constaté que les étudiants en médecine (DCEM2, DCEM3 et DCEM4) obtenaient un score inférieur à celui des internes DES, lui-même inférieur au score des experts. Ce résultat attendu reflète le fait que le TCS évalue davantage que la maîtrise des seules connaissances déclaratives mais également le processus de leur mobilisation dans le cadre du raisonnement clinique et il est en accord avec diffé-rentes études réalisées dans diverses autres spécialités médicales ou chirurgicales [8][9][10] . Par exemple, une série comportant 17 étudiants de DCEM 4, neuf internes du DES de médecine générale, cinq internes inscrits au DES de médecine interne, et sept médecins internistes confirmés, avait montré une bonne discrimination du niveau d'expé-rience clinique par le TCS 8 .…”

Towards a Realistic Description of Competence for New Radiology Graduates in South Africa

“…By clearly stating descriptors of skill competence for perceptual skill, we may isolate indicators of its maturation during training, and ensure that such changes are effected and that ultimately, the curriculum is rendered more efficient. However, despite growing evidence that novice and experts image readers gather, process and retrieve stored mental images differently (Brazeau-Lamontagne, Charlin et al 2004), two assumptions have persisted since 1962. The first is that the mental image is faithful to the external image and the second, that every person gathers visual data in exactly the same manner regardless of the level of expertise (Tuddenham 1962).…”

Assessing intraoperative judgment using script concordance testing through the gynecology continuum of practice

“…These scenarios can be diagnostic, prognostic or therapeutic by design. SCTs have been previous validated in Surgery, Urology, Internal Medicine and OBGYN (Sibert et al 2002;Brazeau-Lamontagne et al 2004;Meterissian 2006;Meterissian et al 2007;Park et al 2010).…”