Harnessing Natural Language Processing to Support Decisions Around Workplace-Based Assessment: Machine Learning Study of Competency-Based Medical Education

Abstract

Background Residents receive a numeric performance rating (eg, 1-7 scoring scale) along with a narrative (ie, qualitative) feedback based on their performance in each workplace-based assessment (WBA). Aggregated qualitative data from WBA can be overwhelming to process and fairly adjudicate as part of a global decision about learner competence. Current approaches with qualitative data require a human rater to maintain attention and appropriately weigh various data inputs within the constraints of working memory before rendering a global judgment of performance. Objective This study explores natural language processing (NLP) and machine learning (ML) applications for identifying trainees at risk using a large WBA narrative comment data set associated with numerical ratings. Methods NLP was performed retrospectively on a complete data set of narrative comments (ie, text-based feedback to residents based on their performance on a task) derived from WBAs completed by faculty members from multiple hospitals associated with a single, large, residency program at McMaster University, Canada. Narrative comments were vectorized to quantitative ratings using the bag-of-n-grams technique with 3 input types: unigram, bigrams, and trigrams. Supervised ML models using linear regression were trained with the quantitative ratings, performed binary classification, and output a prediction of whether a resident fell into the category of at risk or not at risk. Sensitivity, specificity, and accuracy metrics are reported. Results The database comprised 7199 unique direct observation assessments, containing both narrative comments and a rating between 3 and 7 in imbalanced distribution (scores 3-5: 726 ratings; and scores 6-7: 4871 ratings). A total of 141 unique raters from 5 different hospitals and 45 unique residents participated over the course of 5 academic years. When comparing the 3 different input types for diagnosing if a trainee would be rated low (ie, 1-5) or high (ie, 6 or 7), our accuracy for trigrams was 87%, bigrams 86%, and unigrams 82%. We also found that all 3 input types had better prediction accuracy when using a bimodal cut (eg, lower or higher) compared with predicting performance along the full 7-point rating scale (50%-52%). Conclusions The ML models can accurately identify underperforming residents via narrative comments provided for WBAs. The words generated in WBAs can be a worthy data set to augment human decisions for educators tasked with processing large volumes of narrative assessments.

Using Natural Language Processing to Evaluate the Quality of Supervisor Narrative Comments in Competency-Based Medical Education

“…Within medical education, NLP applications have focused primarily on predicting clinical performance. [22][23][24][25] Critically, the few NLP studies focused on evaluating the quality of narrative comments have not incorporated a standardized tool (such as the QuAL score) to define quality. 26,27…”

Finding Medicine’s Moneyball: How Lessons From Major League Baseball Can Advance Assessment in Precision Education

“…28,29 Machine learning algorithms and natural language processing can allow for rapid analysis of qualitative assessment data. 30,31 Clinical care metrics, such as Resident-Sensitive Quality Measures (RSQMs) [32][33][34][35] and Trainee Attributable and Automatable Care Evaluations in Real-time (TRACERs), 36 3). a guiding principle, we need to move from using easily found data to data that truly matter for our learners and patients.…”

“…For example, LA can help make meaning of existing WBA data by seeking to understand how contextual factors such as rater tendencies, rotation order, or time of year affect scores 28,29 . Machine learning algorithms and natural language processing can allow for rapid analysis of qualitative assessment data 30,31 . Clinical care metrics, such as Resident-Sensitive Quality Measures (RSQMs) 32–35 and Trainee Attributable and Automatable Care Evaluations in Real-time (TRACERs), 36 are being developed for trainee assessment using existing electronic health record (EHR) information, creating a new type of assessment data to inform learning.…”

Demystifying AI: Current State and Future Role in Medical Education Assessment

“…Natural language processing (NLP), an application of unsupervised learning, including large language models (LLMs), is an integral part of AI that focuses on the interaction between computers and human language. In medical education assessment, NLP is gaining popularity for its utility in analyzing written content, such as student notes, 15 identifying trends in reflections, 16 or summarizing personalized feedback 17 …”