Validation of a Rubric Used for Skills-Based Assessment of Veterinary Students Performing Simulated Ovariohysterectomy on a Model

Abstract

Surgical skills are an important competency for new graduates. Simulators offer a means to train and assess veterinary students prior to their first surgical performance. A simulated ovariohysterectomy (OVH) rubric’s validity was evaluated using a framework of content evidence, internal structure evidence, and evidence of relationship with other variables, specifically subsequent live surgical performance. Clinically experienced veterinarians ( n = 13) evaluated the utility of each rubric item to collect evidence; each item’s content validity index was calculated to determine its inclusion in the final rubric. After skills training, veterinary students ( n = 57) were assessed using the OVH model rubric in March and August. Internal structure evidence was collected by video-recording 14 students’ mock surgeries, each assessed by all five raters to calculate inter-rater reliability. Relationship with other variables evidence was collected by assessing 22 students performing their first live canine OVH in November. Experienced veterinarians included 22 items in the final rubric. The rubric generated scores with good to excellent internal consistency; inter-rater reliability was fair. Students’ performance on the March model assessment was moderately correlated with their live surgical performance ( ρ = 0.43) and moderately negatively correlated with their live surgical time ( ρ = −0.42). Students’ performance on the August model assessment, after a summer without surgical skills practice, was weakly correlated with their live surgical performance ( ρ = 0.17). These data support validation of the simulated OVH rubric. The continued development of validated assessment instruments is critical as veterinary medicine seeks to become competency based.

Retention of basic surgical skills in veterinary students

“…Raters participated in several rater training sessions over the preceding 2 years; they agreed on the ideal student performance, acceptable variations on technique, and how to score students' errors using the rubric. A previous study reported the reliability of scores produced by the rubric and raters on this assessment; the internal consistency of scores produced by the rubric was good to excellent, and inter‐rater agreement on rubric scores and pass/fail decisions was fair 18 …”

“…Therefore, we cannot predict how quickly and to what extent student performance could rebound following a return to surgical skills training, or to what extent our data may apply to other colleges or settings. We utilized a single surgical skills rubric, which was validated in a previous study and used the same raters, who achieved fair interrater reliability 18 . Assessment of global surgical competence would require multiple assessments of a student performing a variety of surgical procedures, scored using standard, objective, and validated rubrics.…”

“…We utilized a single surgical skills rubric, which was validated in a previous study and used the same raters, who achieved fair interrater reliability. 18 Assessment of global surgical competence would require multiple assessments of a student performing a variety of surgical procedures, scored using standard, objective, and validated rubrics. Not all surgical skills may be subject to the same degree of attrition as demonstrated in this study, and some skills may decay faster than we observed.…”

“…A previous study reported the reliability of scores produced by the rubric and raters on this assessment; the internal consistency of scores produced by the rubric was good to excellent, and inter-rater agreement on rubric scores and pass/fail decisions was fair. 18 Students completed the spring semester, left the university for the summer, and returned to class in the fall. Students participating in the study did no surgical training over the summer break.…”

“…Once enrolled, students participated in the standard second-year surgical skills curriculum in spring 2021, which included 4 Â 2-3 h laboratory sessions to practice performing an OVH on the LMUterus model, which was described previously. 15,18 Upon completion of these laboratories, students were evaluated performing an OVH on their model using a previously published rubric 15 (Table A1 in Appendix 1), which was modified from a rubric created by Read et al 19 The rubric scores students' performance on 22 items using a scale where each item is assigned 3 points for an excellent performance, 2 points for a good performance, 1 point for a borderline performance, and 0 points for an unsatisfactory performance. The maximum possible rubric score was 66 points.…”