Automatic Incident Triage in Radiation Oncology Incident Learning System

Abstract

The Radiotherapy Incident Reporting and Analysis System (RIRAS) receives incident reports from Radiation Oncology facilities across the US Veterans Health Affairs (VHA) enterprise and Virginia Commonwealth University (VCU). In this work, we propose a computational pipeline for analysis of radiation oncology incident reports. Our pipeline uses machine learning (ML) and natural language processing (NLP) based methods to predict the severity of the incidents reported in the RIRAS platform using the textual description of the reported incidents. These incidents in RIRAS are reviewed by a radiation oncology subject matter expert (SME), who initially triages some incidents based on the salient elements in the incident report. To automate the triage process, we used the data from the VHA treatment centers and the VCU radiation oncology department. We used NLP combined with traditional ML algorithms, including support vector machine (SVM) with linear kernel, and compared it against the transfer learning approach with the universal language model fine-tuning (ULMFiT) algorithm. In RIRAS, severities are divided into four categories; A, B, C, and D, with A being the most severe to D being the least. In this work, we built models to predict High (A & B) vs. Low (C & D) severity instead of all the four categories. Models were evaluated with macro-averaged precision, recall, and F1-Score. The Traditional ML machine learning (SVM-linear) approach did well on the VHA dataset with 0.78 F1-Score but performed poorly on the VCU dataset with 0.5 F1-Score. The transfer learning approach did well on both datasets with 0.81 F1-Score on VHA dataset and 0.68 F1-Score on the VCU dataset. Overall, our methods show promise in automating the triage and severity determination process from radiotherapy incident reports.

Documentation of the patient’s smoking status in common chronic diseases – analysis of medical narrative reports using the ULMFiT based text classification

“…In this study, we showed that the performance of the ULMFiT-based algorithm varied from good to excellent in classifying patients’ smoking status from Finnish narrative reports. Overall, ULMFiT and other deep learning-based approaches have shown to be promising tools in standardization of language used in narrative reports including abbreviations, acronyms, eponyms, slang and jargon words [ 18 , 19 ]. An additional benefit of using this type of language model is that, once fine-tuned for Finnish medical narratives, the classifier can be further developed for other study needs.…”

Natural language processing and machine learning to assist radiation oncology incident learning

“…To the best of our knowledge, the only published report of using NLP-ML methods to automate incident learning was by Syed et al in 2020. 38 In their study, they used traditional ML as well as transfer learning methods to perform binary classification of incident severity (high vs. low). Therefore, our work complements the literature by providing first-hand insight into the capability of NLP-ML methods to classify three other radiotherapy inci- 38 reported that the Linear Support Vector Machine (SVM) was the optimal ML model for their radiation oncology incident report data.…”

“…38 In their study, they used traditional ML as well as transfer learning methods to perform binary classification of incident severity (high vs. low). Therefore, our work complements the literature by providing first-hand insight into the capability of NLP-ML methods to classify three other radiotherapy inci- 38 reported that the Linear Support Vector Machine (SVM) was the optimal ML model for their radiation oncology incident report data. This is consistent with our result, as our optimal ML model was the Linear SVR model, which is a regressor derivative of SVM.…”

“…in 2020. 38 In their study, they used traditional ML as well as transfer learning methods to perform binary classification of incident severity (high vs. low). Therefore, our work complements the literature by providing first‐hand insight into the capability of NLP‐ML methods to classify three other radiotherapy incident data elements, namely: the process step of incident occurrence, problem type, and contributing factors.…”

“…Interestingly, Syed et al. 38 reported that the Linear Support Vector Machine (SVM) was the optimal ML model for their radiation oncology incident report data. This is consistent with our result, as our optimal ML model was the Linear SVR model, which is a regressor derivative of SVM.…”