Summative Supervisor Reporting: A Quality Performance Perspective

Abstract

Objective: This study aimed to quality assure Assigned Educational Supervisor (AES) reports, using UK Joint Committee on Surgical Training (JCST) objective criteria, to evaluate contribution Annual Review of Competence Progression (ARCP). Design: Consecutive 145 AES reports from 75 trainers regarding 68 Core Surgical Trainees were assessed from 9 hospitals (2 Tertiary centres (77 reports), 7 District General Hospitals (68 reports)). Reports were assessed by independent assessors based on free text related to performance mapped to curricular objectives, operative logbooks, and Clinical Supervisor (CS) reports, and overall summary grades assigned ranging from development required, adequate, good, to excellent. Setting: A core surgical training programme serving a single UK (Wales) deanery. Participants: Sixty-eight consecutively appointed core surgical trainees and seventy-five consultant surgeon trainers. Results: Summary grades of adequate or above were achieved in 101/145 (69.7%) reports. Trainees' objective setting meetings were completed within 6 weeks of starting placements in 124/145 (85.5%). The proportions of AES reports containing free text commentary on curricular objectives, portfolio objectives, and operative logbook development were 128/145, 123/145, and 55/145 respectively. AES report quality was not associated with hospital status, subspecialty, or trainee grade. Female trainers were significantly more likely to provide reports graded as Good or Excellent compared with their male colleagues (7/12 vs. 27/133,  2 (2) = 9.389, p= 0.009). AES reports for male trainees were significantly more likely to be rated as further development required (40/85, 47.1%) when compared with female trainees (4/32, 12.5%, p=0.007). 3 Conclusion: Three in ten AES reports were insufficient to contribute to objective ARCP outcomes and a gender gap was apparent related to engagement. AES trainers should provide more focus if this summative tool is to be an effective career progression metric.

Being a good educational supervisor

“…24 Unfortunately, a study of the quality of ES reports to the ARCP demonstrated that less than 30% were 'satisfactory'. 25 The main issues are related to failure to review all the evidence (specifically the logbooks and comments from clinical supervisors) and failure to use the free text tools to expand on or justify the recommendation given.…”