Ontological attention ensembles for capturing semantic concepts in ICD code prediction from clinical text

Falis, Matúš; Pajak, Maciej; Lisowska, Aneta; Schrempf, Patrick; Deckers, Lucas; Mikhael, Shadia; Tsaftaris, Sotirios A.; O’Neil, Alison Q.

doi:10.18653/v1/d19-6220

Cited by 21 publications

(20 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To alleviate the burden of the status quo of manual coding, several Machine learning (ML) automated coding models have been developed (Larkey and Croft, 1996;Aronson et al, 2007;Farkas and Szarvas, 2008;Perotte et al, 2014;Ayyar et al, 2016;Baumel et al, 2018;Mullenbach et al, 2018;Falis et al, 2019). However, despite continued interest, translation of ML systems into real-world deployments has been limited.…”

Section: Introductionmentioning

confidence: 99%

Experimental Evaluation and Development of a Silver-Standard for the MIMIC-III Clinical Coding Dataset

Searle¹,

Ibrahim²,

Dobson³

2020

Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing

View full text Add to dashboard Cite

Clinical coding is currently a labour-intensive, error-prone, but critical administrative process whereby hospital patient episodes are manually assigned codes by qualified staff from large, standardised taxonomic hierarchies of codes. Automating clinical coding has a long history in NLP research and has recently seen novel developments setting new state of the art results. A popular dataset used in this task is MIMIC-III, a large intensive care database that includes clinical free text notes and associated codes. We argue for the reconsideration of the validity MIMIC-III's assigned codes that are often treated as gold-standard, especially when MIMIC-III has not undergone secondary validation. This work presents an open-source, reproducible experimental methodology for assessing the validity of codes derived from EHR discharge summaries. We exemplify the methodology with MIMIC-III discharge summaries and show the most frequently assigned codes in MIMIC-III are under-coded up to 35%.

show abstract

Section: Introductionmentioning

confidence: 99%

Experimental Evaluation and Development of a Silver-Standard for the MIMIC-III Clinical Coding Dataset

Searle¹,

Ibrahim²,

Dobson³

2020

Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing

View full text Add to dashboard Cite

show abstract

“…Out of the 256 publications, we excluded 65 publications, as the described Natural Language Processing algorithms in those publications were not evaluated. The full text of the remaining 191 publications was assessed and 114 a As reported by authors [11,30,33,35,37,41,44,46,49,55,58,64,68,69,72,73,75,76,83,87,90,95,98,100,104] Included reference to dataset 21 (27%) [11,30,35,37,41,44,46,49,55,58,64,72,75,76,83,87,90,95,98,100,104] Training of algorithm Trained 47 (61%) [11, 12, 29, 31, 32, 34, 37, 39, 41, 42, 44, 45, 48-53, 55-59, 62, 63, 65, 66, 68, 69, 73, 74, 76, 78-84, 87, 88, 90, 95, 96, 98, 99, 104] Not listed 3 (3.9%) …”

Section: Resultsmentioning

confidence: 99%

“…Use of development set 16 (21%) [12,29,31,34,37,49,55,60,63,69,74,80,87,90,94,95] Not listed 4 (5.2%) [30,82,83,101] publications did not meet our criteria, of which 3 publications in which the algorithm was not evaluated, resulting in 77 included articles describing 77 studies. Reference checking did not provide any additional publications.…”

Section: Development Of Algorithmmentioning

confidence: 99%

Natural language processing algorithms for mapping clinical text fragments onto ontology concepts: a systematic review and recommendations for future studies

et al. 2020

View full text Add to dashboard Cite

Background Free-text descriptions in electronic health records (EHRs) can be of interest for clinical research and care optimization. However, free text cannot be readily interpreted by a computer and, therefore, has limited value. Natural Language Processing (NLP) algorithms can make free text machine-interpretable by attaching ontology concepts to it. However, implementations of NLP algorithms are not evaluated consistently. Therefore, the objective of this study was to review the current methods used for developing and evaluating NLP algorithms that map clinical text fragments onto ontology concepts. To standardize the evaluation of algorithms and reduce heterogeneity between studies, we propose a list of recommendations. Methods Two reviewers examined publications indexed by Scopus, IEEE, MEDLINE, EMBASE, the ACM Digital Library, and the ACL Anthology. Publications reporting on NLP for mapping clinical text from EHRs to ontology concepts were included. Year, country, setting, objective, evaluation and validation methods, NLP algorithms, terminology systems, dataset size and language, performance measures, reference standard, generalizability, operational use, and source code availability were extracted. The studies’ objectives were categorized by way of induction. These results were used to define recommendations. Results Two thousand three hundred fifty five unique studies were identified. Two hundred fifty six studies reported on the development of NLP algorithms for mapping free text to ontology concepts. Seventy-seven described development and evaluation. Twenty-two studies did not perform a validation on unseen data and 68 studies did not perform external validation. Of 23 studies that claimed that their algorithm was generalizable, 5 tested this by external validation. A list of sixteen recommendations regarding the usage of NLP systems and algorithms, usage of data, evaluation and validation, presentation of results, and generalizability of results was developed. Conclusion We found many heterogeneous approaches to the reporting on the development and evaluation of NLP algorithms that map clinical text to ontology concepts. Over one-fourth of the identified publications did not perform an evaluation. In addition, over one-fourth of the included studies did not perform a validation, and 88% did not perform external validation. We believe that our recommendations, alongside an existing reporting standard, will increase the reproducibility and reusability of future studies and NLP algorithms in medicine.

show abstract

“…ICD coding vs. outcome prediction. There is a variety of work in the related field of automated ICD coding (Xie et al, 2018;Falis et al, 2019). Zhang et al ( 2020) recently presented a model able to identify up to 2,292 ICD codes from text.…”

Section: Related Workmentioning

confidence: 99%

Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integration

Aken¹,

Papaioannou²,

Mayrdorfer³

et al. 2021

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

View full text Add to dashboard Cite

Outcome prediction from clinical text can prevent doctors from overlooking possible risks and help hospitals to plan capacities. We simulate patients at admission time, when decision support can be especially valuable, and contribute a novel admission to discharge task with four common outcome prediction targets: Diagnoses at discharge, procedures performed, in-hospital mortality and length-of-stay prediction. The ideal system should infer outcomes based on symptoms, pre-conditions and risk factors of a patient. We evaluate the effectiveness of language models to handle this scenario and propose clinical outcome pretraining to integrate knowledge about patient outcomes from multiple public sources. We further present a simple method to incorporate ICD code hierarchy into the models. We show that our approach improves performance on the outcome tasks against several baselines. A detailed analysis reveals further strengths of the model, including transferability, but also weaknesses such as handling of vital values and inconsistencies in the underlying data.PRESENT ILLNESS: 58yo man w/ hx of hypertension, AFib on coumadin and NIDDM presented to ED with the worst headache of his life. He had a syncopal episode and was intubated by EMS. Medication on admission: 1mg IV ativan x 1.

show abstract

Ontological attention ensembles for capturing semantic concepts in ICD code prediction from clinical text

Cited by 21 publications

References 10 publications

Experimental Evaluation and Development of a Silver-Standard for the MIMIC-III Clinical Coding Dataset

Experimental Evaluation and Development of a Silver-Standard for the MIMIC-III Clinical Coding Dataset

Natural language processing algorithms for mapping clinical text fragments onto ontology concepts: a systematic review and recommendations for future studies

Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integration

Contact Info

Product

Resources

About