Detection of Residents With Progress Issues Using a Keyword–Specific Algorithm

Abstract

Background The literature suggests that specific keywords included in summative rotation assessments might be an early indicator of abnormal progress or failure. Objective This study aims to determine the possible relationship between specific keywords on in-training evaluation reports (ITERs) and subsequent abnormal progress or failure. The goal is to create a functional algorithm to identify residents at risk of failure. Methods A database of all ITERs from all residents training in accredited programs at Université Laval between 2001 and 2013 was created. An instructional designer reviewed all ITERs and proposed terms associated with reinforcing and underperformance feedback. An algorithm based on these keywords was constructed by recursive partitioning using classification and regression tree methods. The developed algorithm was tuned to achieve 100% sensitivity while maximizing specificity. Results There were 41 618 ITERs for 3292 registered residents. Residents with failure to progress were detected for family medicine (6%, 67 of 1129) and 36 other specialties (4%, 78 of 2163), while the positive predictive values were 23.3% and 23.4%, respectively. The low positive predictive value may be a reflection of residents improving their performance after receiving feedback or a reluctance by supervisors to ascribe a “fail” or “in difficulty” score on the ITERs. Conclusions Classification and regression trees may be helpful to identify pertinent keywords and create an algorithm, which may be implemented in an electronic assessment system to detect future residents at risk of poor performance.

Workplace‐based Assessment Data in Emergency Medicine: A Scoping Review of the Literature

“…In this capacity, it would be valuable to utilize keyword technology and machine learning algorithms to identify trends to facilitate earlier interventions among "at-risk" learners. 33,65 More recently (and outside of our present study period), a conversation has begun in the literature that describes the nature of dashboards that marshal these types of real-time data flows. 19,66 Exciting new areas of research that have more recently shown up in the literature (both within and outside of EM) include the incorporation of EHR data for trending or tracking workplace outcomes.…”

“…Data should be aggregated and presented in a manner to augment decision making or trainee support in a meaningful format. In this capacity, it would be valuable to utilize keyword technology and machine learning algorithms to identify trends to facilitate earlier interventions among “at‐risk” learners 33,65 . More recently (and outside of our present study period), a conversation has begun in the literature that describes the nature of dashboards that marshal these types of real‐time data flows 19,66 .…”

Warnings in early narrative assessment that might predict performance in residency: signal from an internal medicine residency program

“…Estimates suggest anywhere from 5 to 10% of residents "struggle" during their training, with many problems identified too late to maximize help to the learner [14][15][16][17]. Multiple studies have shown the value of narrative data in making summative decisions, while others have attempted to predict ongoing struggles with learner development [12,[18][19][20]. However, challenges exist when using narrative data in these ways.…”

“…Carefully collected and analyzed narrative assessment data can provide value, particularly early in training given their potential to capture rich details about a learner’s performance. Different approaches to finding value in narrative data have included the use of “keyword algorithms” or counting the number of words and percentage of assessments containing negative or ambiguous comments, which were associated with the need for remediation [ 15 , 19 ]. However, using keyword algorithms was much better at ruling out, rather than predicting who would struggle as evidenced by a positive predictive value of 23% [ 19 ].…”

“…Different approaches to finding value in narrative data have included the use of “keyword algorithms” or counting the number of words and percentage of assessments containing negative or ambiguous comments, which were associated with the need for remediation [ 15 , 19 ]. However, using keyword algorithms was much better at ruling out, rather than predicting who would struggle as evidenced by a positive predictive value of 23% [ 19 ]. Using negative or ambiguous comments was helpful when reading an entire residency file, but 12 of 17 residents in good standing still had negative comments [ 15 ].…”