Natural language processing algorithms for mapping clinical text fragments onto ontology concepts: a systematic review and recommendations for future studies

Abstract

Background Free-text descriptions in electronic health records (EHRs) can be of interest for clinical research and care optimization. However, free text cannot be readily interpreted by a computer and, therefore, has limited value. Natural Language Processing (NLP) algorithms can make free text machine-interpretable by attaching ontology concepts to it. However, implementations of NLP algorithms are not evaluated consistently. Therefore, the objective of this study was to review the current methods used for developing and evaluating NLP algorithms that map clinical text fragments onto ontology concepts. To standardize the evaluation of algorithms and reduce heterogeneity between studies, we propose a list of recommendations. Methods Two reviewers examined publications indexed by Scopus, IEEE, MEDLINE, EMBASE, the ACM Digital Library, and the ACL Anthology. Publications reporting on NLP for mapping clinical text from EHRs to ontology concepts were included. Year, country, setting, objective, evaluation and validation methods, NLP algorithms, terminology systems, dataset size and language, performance measures, reference standard, generalizability, operational use, and source code availability were extracted. The studies’ objectives were categorized by way of induction. These results were used to define recommendations. Results Two thousand three hundred fifty five unique studies were identified. Two hundred fifty six studies reported on the development of NLP algorithms for mapping free text to ontology concepts. Seventy-seven described development and evaluation. Twenty-two studies did not perform a validation on unseen data and 68 studies did not perform external validation. Of 23 studies that claimed that their algorithm was generalizable, 5 tested this by external validation. A list of sixteen recommendations regarding the usage of NLP systems and algorithms, usage of data, evaluation and validation, presentation of results, and generalizability of results was developed. Conclusion We found many heterogeneous approaches to the reporting on the development and evaluation of NLP algorithms that map clinical text to ontology concepts. Over one-fourth of the identified publications did not perform an evaluation. In addition, over one-fourth of the included studies did not perform a validation, and 88% did not perform external validation. We believe that our recommendations, alongside an existing reporting standard, will increase the reproducibility and reusability of future studies and NLP algorithms in medicine.

Case Report: Utilizing AI and NLP to Assist with Healthcare and Rehabilitation During the COVID-19 Pandemic

“…The success of AI/ML in the image analysis domain can be attributed to the wide availability of high quality, comprehensive, and extensively annotated datasets. In other domains, such as NLP processing of electronic health records, there is an absence of publically available annotated datasets which can be used to develop and validate NLP systems ( Kersloot et al, 2020 ). Due to this, there is limited information about the accuracy of NLP healthcare data analysis systems within the literature and it is difficult to compare the existing systems within the research ( Kersloot et al, 2020 ).…”

“…In other domains, such as NLP processing of electronic health records, there is an absence of publically available annotated datasets which can be used to develop and validate NLP systems ( Kersloot et al, 2020 ). Due to this, there is limited information about the accuracy of NLP healthcare data analysis systems within the literature and it is difficult to compare the existing systems within the research ( Kersloot et al, 2020 ). The development of publicly available challenge NLP healthcare datasets and better metrics for analyzing the accuracy of such systems is an area which should be worked on by researchers in the future.…”

Citywide quality of health information system through text mining of electronic health records

“…Sun et al present the architecture of their semantic processing approach where data is transmitted through the semantic layer with clinical ontologies inside (Sun et al 2015). Most solutions for data interoperability are based on ontologies (Sun et al 2015;Roberts and Demner-Fushman 2016;Freedman et al 2020;Kersloot et al 2020). Moreover, Kersloot et al (2020) show the statistics for mapping clinical text fragments to ontology concepts that are described in reviewed papers.…”

“…Most solutions for data interoperability are based on ontologies (Sun et al 2015;Roberts and Demner-Fushman 2016;Freedman et al 2020;Kersloot et al 2020). Moreover, Kersloot et al (2020) show the statistics for mapping clinical text fragments to ontology concepts that are described in reviewed papers. They conclude that 88% of the studies do not present any validation.…”

Government plans in the 2016 and 2021 Peruvian presidential elections: A natural language processing analysis of the health chapters

“…Over the last few years, researchers working in clinical medicine have adopted artificial intelligence and deep learning techniques such as Natural Language Processing (NLP). Due to this, electronic health records have become unique data sources because they contain free text annotations that can inform NLP models for a variety of tasks [1][2][3][4][5][6][7] (e.g., risk prediction). On the other hand, public health research appears not to have benefited from NLP algorithms, despite the fact that this research field also has large data sources of text, such as reports and policy briefs 8 .…”