Can Unified Medical Language System–based semantic representation improve automated identification of patient safety incident reports by type and severity?

Abstract

Objective The study sought to evaluate the feasibility of using Unified Medical Language System (UMLS) semantic features for automated identification of reports about patient safety incidents by type and severity. Materials and Methods Binary support vector machine (SVM) classifier ensembles were trained and validated using balanced datasets of critical incident report texts (n_type = 2860, n_severity = 1160) collected from a state-wide reporting system. Generalizability was evaluated on different and independent hospital-level reporting system. Concepts were extracted from report narratives using the UMLS Metathesaurus, and their relevance and frequency were used as semantic features. Performance was evaluated by F-score, Hamming loss, and exact match score and was compared with SVM ensembles using bag-of-words (BOW) features on 3 testing datasets (type/severity: n_benchmark = 286/116, n_original = 444/4837, n_independent =6000/5950). Results SVMs using semantic features met or outperformed those based on BOW features to identify 10 different incident types (F-score [semantics/BOW]: benchmark = 82.6%/69.4%; original = 77.9%/68.8%; independent = 78.0%/67.4%) and extreme-risk events (F-score [semantics/BOW]: benchmark = 87.3%/87.3%; original = 25.5%/19.8%; independent = 49.6%/52.7%). For incident type, the exact match score for semantic classifiers was consistently higher than BOW across all test datasets (exact match [semantics/BOW]: benchmark = 48.9%/39.9%; original = 57.9%/44.4%; independent = 59.5%/34.9%). Discussion BOW representations are not ideal for the automated identification of incident reports because they do not account for text semantics. UMLS semantic representations are likely to better capture information in report narratives, and thus may explain their superior performance. Conclusions UMLS-based semantic classifiers were effective in identifying incidents by type and extreme-risk events, providing better generalizability than classifiers using BOW.

Risk Management and Patient Safety in the Artificial Intelligence Era: A Systematic Review

“…Due to the high volume of data collected by the IT systems responsible for these purposes, using free text for reporting these events reduces its effectiveness due to the difficulty in aggregating the data [ 86 ]. Of the 36 articles included in this systematic review, 13 [ 38 , 39 , 40 , 41 , 42 , 43 , 44 , 47 , 48 , 49 , 50 , 51 , 53 ] investigate the applications of AI to improve the efficiency of incident reporting systems. The studies predominantly focus their attention on the possibility of standardizing events according to their type and severity.…”

Automating the Identification of Safety Events Involving Machine Learning-Enabled Medical Devices

“…Empty reports were excluded, and text was changed into lower case. To provide informative features for classification, word tokenization, removal of stop words, stemming and lemmatization were applied [8].…”

“…Text classifiers driven by ML methods have been shown to be feasible for identifying events about a variety of patient safety problems such as falls and medication errors [7][8][9][10]. We have previously demonstrated the feasibility of using text classification to identify safety events involving health IT from MAUDE [11].…”

The UMLS knowledge sources at 30: indispensable to current research and applications in biomedical informatics

“…31 Wang et al determined that use of the UMLS could improve automated categorization of patient safety incident reports. 32 Rasmy et al found the UMLS useful in representing electronic health record (EHR) data in predictive modeling. 33 Bitton et al mapped transliterated terms to UMLS concepts to improve retrieval in a Hebrew online health community, 34 an example of using the UMLS both to extract information from social media and to aid interpretation of non-English text.…”