Education Research: The Narrative Evaluation Quality Instrument

Abstract

ObjectiveDetermining the quality of narrative evaluations to assess medical student neurology clerkship performance remains a challenge. This study sought to develop a tool to comprehensively and systematically assess quality of student narrative evaluations.MethodsThe Narrative Evaluation Quality Instrument (NEQI) was created to assess several components within clerkship narrative evaluations: performance domains, specificity, and usefulness to learner. In this retrospective study, 5 investigators scored 123 narrative evaluations using the NEQI. Inter-rater reliability was estimated by calculating interclass correlation coefficients (ICC) across 615 NEQI scores.ResultsThe average overall NEQI score was 6.4 (SD 2.9), with mean component arm scores of 2.6 for performance domains (SD 0.9), 1.8 for specificity (SD 1.1), and 2.0 for usefulness (SD 1.4). Each component arm exhibited moderate reliability: performance domains ICC 0.65 (95% confidence interval [CI] 0.58–0.72), specificity ICC 0.69 (95% CI 0.61–0.77), and usefulness ICC 0.73 (95% CI 0.66–0.80). Overall NEQI score exhibited good reliability (0.81; 95% CI 0.77–0.86).ConclusionThe NEQI is a novel, reliable tool to comprehensively assess the quality of narrative evaluation of neurology clerks and will enhance the study of interventions seeking to improve clerkship evaluation.

Education Research: A Long-term Faculty Development Initiative Improves Specificity and Usefulness of Narrative Evaluations of Clerkship Students

“…22,25,27 Such interventions, particularly when incorporating process-level and structural-level changes to affect the broader learning environment and culture around trainee assessment, are poised to facilitate skill improvement and mastery. Although prior work with the NEQI has established several sources of validity evidence including that regarding content and internal structure, 14,29 this study provides additional evidence relating to consequences 41 by documenting the intended effect of the intervention through an improvement in the quality of narrative assessments. It is important that no unintended effects of the NEQI scores were observed based by removing numeric scores from the clerkship ITERs.…”

“…With respect to the observed effect, the relative difference in the control and intervention groups might appear modest; however, our effect is similar in magnitude to prior studies that have shown improvements in assessments, including written comments, after rater/assessor training. 27,39,40 In addition, our prior work developing the NEQI 29 has suggested that an NEQI score of 7 represents a minimum quality threshold, with the bulk of evaluations in the historical control group failing to reach this level, although further work is needed to understand whether this threshold is of educational significance.…”

“…Departmental resources were critical to developing and embedding these efforts into our education program and conveying a collective commitment to improving trainee assessment. Although prior work with the NEQI has established several sources of validity evidence including content and internal structure, 14,29 future work will collect examine consequences evidence 30 by how examining trainees and promotion committees may differentially interpret and use higher-scored vs lower-scored comments using the NEQI. Future research will also involve identifying specific assessor-level factors that are associated with overall and subscale NEQI scores and examining the effect of providing individualized feedback, rather than aggregate group feedback, on narrative quality.…”

“…The NEQI assesses the quality of narrative evaluations along several dimensions including performance domains, specificity of comments, and usefulness to trainees and has been shown to reliably differentiate between narrative quality. 29 Before narrative evaluation review, 4 reviewers (authors: S.J.P., S.D., C.E., and B.R.) used the NEQI training guide alongside 12 student evaluations for training purposes.…”

“…We hypothesized that such an effort would demonstrate higher-quality narratives in the intervention group compared with a historical control group, as measured by the narrative evaluation quality instrument (NEQI), which comprehensively assesses the quality of narrative evaluations. 29 As a secondary aim, we sought to collect additional validity evidence 30 of the NEQI.…”