Automatic rubric-based content grading for clinical notes

Abstract

Clinical notes provide important documentation critical to medical care, as well as billing and legal needs. Too little information degrades quality of care; too much information impedes care. Training for clinical note documentation is highly variable, depending on institutions and programs. In this work, we introduce the problem of automatic evaluation of note creation through rubric-based content grading, which has the potential for accelerating and regularizing clinical note documentation training. To this end, we describe our corpus creation methods as well as provide simple feature-based and neural network baseline systems. We further provide tagset and scaling experiments to inform readers of plausible expected performances. Our baselines show promising results with content point accuracy and kappa values at 0.86 and 0.71 on the test set.

Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI)

“…Despite its importance, the task of automatic grading of patient notes remains under-explored with only a few works that have studied it (Yim et al, 2019;Sarker et al, 2019). Traditional supervised models have been utilized for this task (Latifi et al, 2016;Yim et al, 2019), but are limited in scope because they rely on large scale annotated datasets. The significant manual effort associated with labeled dataset creation makes these methods difficult and impractical.…”

“…While there are some works on ASAG for scientific topics, only three works studied automatic patient note assessment (Latifi et al, 2016;Sarker et al, 2019;Yim et al, 2019). Inspired by the works on ASAG, the first two (Latifi et al, 2016;Yim et al, 2019) studied two systems: a feature based system including an n-gram feature extraction followed by a SVM and a simple BERT based neural network. The third (Sarker et al, 2019) followed previous works on ASAG and leveraged the pipeline framework.…”

Zero-Shot Multimodal Question Answering for Assessment of Medical Student OSCE Physical Exam Videos

“…To date, most success has been found in developing systems that grade post-encounter notes, the free-text responses that students write after the simulated encounter. Recent attempts have focused on using supervised learning (Sarker et al, 2019; Yim et al, 2019) that utilize previously scored triples of student responses, rubrics, and scoring data to train binary classifiers or semi-supervised approaches that use clinical entity recognition pipelines and curated acceptable answer lists to test for the presence of key phrases (Bond et al, 2023).…”