The educational value of using cumulative sum charts

Abstract

SummaryVarious workplace-based assessment tools are available, but none have been shown to improve performance in procedural skills. This study aimed to assess the impact of using one such tool, cumulative sum charts, on procedural skill ability. A single-blind randomised controlled trial was conducted on 82 final year medical students. Control group students received the usual teaching; in addition to this, intervention group students were provided with cumulative sum charts to log their cannulation attempts over a 7-month period. At the end of the year, students from both groups undertook a validated test of automaticity of cannulation skill. Students in the intervention group obtained median (IQR [range]) scores of 68.2 (60.5-74.3 [42.7-81.1]) vs 62.2 (52. 2-68.8 [40.7-80.5]) for the control group (p = 0.013). The effect size was moderate (Cohen's d = 0.608). This study therefore provides support for the hypothesis that use of cumulative sum charts improves performance when learning procedural skills. All UK specialty training schemes utilise a variety of workplace-based assessment tools. Advantages of such tools include the fact that they have limited impact on service delivery, they are relatively cost-effective, and they assess what a trainee actually does rather than what they know how to do. A well-established method of evaluating workplace-based assessment tools is van der Vleuten's framework, which includes reliability, validity, acceptability, cost and educational impact [1]. The relative importance of these five factors depends upon whether the tool is to be used formatively (assessment for learning) or summatively (assessment of learning) [1]. In summative assessments, reliability and validity are crucially important, whereas in formative assessments, the educational impact of a tool is paramount.An example of a formative workplace-based assessment tool is the cumulative sum (CUSUM) chart. Cumulative sum charts, originally used to monitor the quality of manufactured products [2], have been used to monitor learning in many specialties, including anaesthesia [3], cardiac surgery [4], gynaecology [5] and midwifery [6]. There are many variants of CUSUM charts, including the learning curve CUSUM [7] and the risk-adjusted CUSUM [8].In this study, we examined a simple CUSUM chart, as described by de Oliveira Filho in 2002 [3]. This type of chart can be used for any skill in which a binary Anaesthesia 2012Anaesthesia , 67, 734-740 doi:10.1111Anaesthesia /j.1365Anaesthesia -2044Anaesthesia .2012 ). An example of a CUSUM chart and the formulae used are shown in Fig. 1. To our knowledge, there have been no studies that assess the value of using CUSUM charts as a formative assessment tool. There are, however, educational theories that suggest that their use might result in improved performance. It has been suggested that self-regulation tools that provide explicit performance standards may improve self-assessment [9]. It also seems plausible that a chart that encourages help-seeking behaviour or further p...

Basic practical skills teaching and learning in undergraduate medical education – a review on methodological evidence

“…In addition, mental imagery technique appeared to transfer learning better from practice suturing sessions to actual surgical assessment than textbook study [48]. Better cannulation skills could be demonstrated by students using cumulative sum charts to log their cannulation attempts during their finial year [51]. When students were taught gastric-tube insertion with Peyton’s Four-Step Approach as teaching method they did not differ from their peers, who received standard instruction in terms of correct stepwise performance but sored better in global rating assessing professionalism [25].…”

Challenges & Issues: Evidence-Based Clinical Skills Teaching and Learning: What Do We Really Know?

“…67 As an additional benefit, individual learning curve charts improved the intravenous cannulation ability of medical students, likely through improved self-regulation. 68 Learning curves have also been suggested to play a role in identifying milestones for required competencies, identifying struggling learners or ineffective training programs, and identifying training of limited benefit. Group learning curve charts may be helpful for estimating the average time required for skill acquisition and for comparing interventions and training programs.…”

Calibrated breast density methods for full field digital mammography: A system for serial quality control and inter‐system generalization

“…Since its original development, Cusum has been used in various arenas including medical outcomes evaluation and public health surveillance. Such applications include monitoring performance of clinicians in various specialties, [21][22][23][24][25][26][27][28][29][30] patient survival following transplant procedures, 31 screening process quality, 32 infectious disease outbreaks, [33][34][35][36] cancer incidence, 37 air pollution levels, 38 and occupational safety events. 39 Our calibration methodology was initially developed using indirect x-ray conversion full field digital mammography (FFDM) technology.…”