Concept recognition as a machine translation problem

Abstract

Background Automated assignment of specific ontology concepts to mentions in text is a critical task in biomedical natural language processing, and the subject of many open shared tasks. Although the current state of the art involves the use of neural network language models as a post-processing step, the very large number of ontology classes to be recognized and the limited amount of gold-standard training data has impeded the creation of end-to-end systems based entirely on machine learning. Recently, Hailu et al. recast the concept recognition problem as a type of machine translation and demonstrated that sequence-to-sequence machine learning models have the potential to outperform multi-class classification approaches. Methods We systematically characterize the factors that contribute to the accuracy and efficiency of several approaches to sequence-to-sequence machine learning through extensive studies of alternative methods and hyperparameter selections. We not only identify the best-performing systems and parameters across a wide variety of ontologies but also provide insights into the widely varying resource requirements and hyperparameter robustness of alternative approaches. Analysis of the strengths and weaknesses of such systems suggest promising avenues for future improvements as well as design choices that can increase computational efficiency with small costs in performance. Results Bidirectional encoder representations from transformers for biomedical text mining (BioBERT) for span detection along with the open-source toolkit for neural machine translation (OpenNMT) for concept normalization achieve state-of-the-art performance for most ontologies annotated in the CRAFT Corpus. This approach uses substantially fewer computational resources, including hardware, memory, and time than several alternative approaches. Conclusions Machine translation is a promising avenue for fully machine-learning-based concept recognition that achieves state-of-the-art results on the CRAFT Corpus, evaluated via a direct comparison to previous results from the 2019 CRAFT shared task. Experiments illuminating the reasons for the surprisingly good performance of sequence-to-sequence methods targeting ontology identifiers suggest that further progress may be possible by mapping to alternative target concept representations. All code and models can be found at: https://github.com/UCDenver-ccp/Concept-Recognition-as-Translation.

Creating an Ignorance-Base: Exploring Known Unknowns in the Scientific Literature

“…We combine the best-performing ignorance classifiers (extending the work of [82] to create a corpus of 91 articles) with state-of-the-art biomedical concept classifiers [117] to create an ignorance-base that allows us to explore it by a topic and by experimental results. The ignorance-base can be queried by ignorance category, specific lexical cues, ontology concepts, or any combination of the three.…”

“…The continued annotation effort used Knowtator [129] and Protege [130] as in previous work [82], allowing the ignorance taxonomy (see Table 2) to be easily browsable like an ontology, and helping the annotators select the correct level of specificity for each lexical cue. The classification frameworks and models were also from our previous work [117,82].…”

“…All models were chosen based on our prior work [82] and on an evaluation of many canonical models for concept recognition [117]. They found that the CRF [137] and BioBERT [138] performed the best for span detection [117]. For sentence classification, we began with the simple ANN from our prior work and added both BERT [139] and BioBERT [138], as they are state-of-the-art language models.…”

“…In order to also understand the biomedical subjects of these statements and be able to connect them to other knowledge-bases, we identified the biomedical concepts from the open biomedical ontologies (OBOs) (e.g. the all caps words in the example sentence) using concept recognition methods [117]. The ignorance-base combines the identification and classification of ignorance statements and biomedical concept recognition.…”

A Gated Recurrent Unit based architecture for recognizing ontology concepts from biological literature

“…Most recent publications in this area have separated the ontology annotation task to two sub-tasks -1) span detection: detecting the part of text that corresponds to an ontology concept, and 2) concept normalization: identifying the ontology concept most appropriate for the identified piece of text [27,28]. Using the CRAFT corpus as a training set, the study reports that Bidirectional encoder representations from transformers for biomedical text mining (BioBERT) resulted in the best performance (0.81 F1) for the span detection sub-task.…”