Modelling the pre‐assessment learning effects of assessment: evidence in the validity chain

Abstract

OBJECTIVESWe previously developed a model of the pre-assessment learning effects of consequential assessment and started to validate it. The model comprises assessment factors, mechanism factors and learning effects. The purpose of this study was to continue the validation process. For stringency, we focused on a subset of assessment factor–learning effect associations that featured least commonly in a baseline qualitative study. Our aims were to determine whether these uncommon associations were operational in a broader but similar population to that in which the model was initially derived.METHODSA cross-sectional survey of 361 senior medical students at one medical school was undertaken using a purpose-made questionnaire based on a grounded theory and comprising pairs of written situational tests. In each pair, the manifestation of an assessment factor was varied. The frequencies at which learning effects were selected were compared for each item pair, using an adjusted alpha to assign significance. The frequencies at which mechanism factors were selected were calculated.RESULTSThere were significant differences in the learning effect selected between the two scenarios of an item pair for 13 of this subset of 21 uncommon associations, even when a p-value of < 0.00625 was considered to indicate significance. Three mechanism factors were operational in most scenarios: agency; response efficacy, and response value.CONCLUSIONSFor a subset of uncommon associations in the model, the role of most assessment factor–learning effect associations and the mechanism factors involved were supported in a broader but similar population to that in which the model was derived. Although model validation is an ongoing process, these results move the model one step closer to the stage of usefully informing interventions. Results illustrate how factors not typically included in studies of the learning effects of assessment could confound the results of interventions aimed at using assessment to influence learning.Discuss ideas arising from this article at ‘http://www.mededuc.com discuss’

Clinical examination in the OSCE era: are we maintaining the balance between OS and CE?

“…As recently stated by Harden,3 more than 1600 articles have been published on OSCEs, and one quarter of those have been published since 2011. However, very few studies have focused on the impact of the ‘OSCE era’ on how students learn and perform clinical examinations 1 2 4. In agreement with Hodges,5 we believe that OSCEs may have shaped the roles doctors play in clinical practice, an area worth investigating.…”

“…The task demands of OSCEs and the importance of the examination exert a strong influence on students 1 2. Therefore, the pre- and post-assessment effects of OSCEs on students' learning may take precedence over the demands of other assessment methods.…”

“…For instance, mini-clinical evaluation exercises (Mini-CEX), Entrustable Professional Activities (EPA) and video recordings of actual practice are valid methods currently used in the workplace. Awaiting clarification studies, we can expect students to redirect their focus on CE rather than OS in those whole-task assessments 1 6 7. Consequently, a combination of assessments, where trainees are actually performing a clinical examination instead of only ‘showing how’, will help maintain the balance between OS and CE 10…”

Medical school benchmarking – From tools to programmes

“…are also important as assessment of students is a powerful driver of learning behaviour (Newble & Jaeger 1983;Frederiksen 1984;Wilkinson et al 2007;Cilliers et al 2010Cilliers et al , 2012aCilliers et al , 2012bCilliers et al , 2013 and therefore we argue it is important to theorise and understand how benchmarking of medical schools could drive organisational behaviours. Such behaviours could relate to curriculum development activities and other aspects of programme delivery.…”

“…(5) What is the likely ''institutional impact'' of the results? The ''educational impact'' of assessment on student learning behaviour is well known (Newble & Jaeger 1983;Frederiksen 1984;Wilkinson et al 2007;Cilliers et al 2010Cilliers et al , 2012aCilliers et al , 2012bCilliers et al , 2013. For example, assessment that focuses only on rote factual knowledge, and not on communication, will lead students to memorising facts and not learning communication skills as well.…”