HunFlair2 in a cross-corpus evaluation of biomedical named entity recognition and normalization tools

Abstract

Motivation With the exponential growth of the life sciences literature, biomedical text mining (BTM) has become an essential technology for accelerating the extraction of insights from publications. The identification of entities in texts, such as diseases or genes, and their normalization, i.e. grounding them in knowledge base, are crucial steps in any BTM pipeline to enable information aggregation from multiple documents. However, tools for these two steps are rarely applied in the same context in which they were developed. Instead, they are applied “in the wild”, i.e. on application-dependent text collections from moderately to extremely different from those used for training, varying e.g. in focus, genre or text type. This raises the question whether the reported performance, usually obtained by training and evaluating on different partitions of the same corpus, can be trusted for downstream applications. Results Here, we report on the results of a carefully designed cross-corpus benchmark for entity recognition and normalization, where tools were applied systematically to corpora not used during their training. Based on a survey of 28 published systems, we selected five, based on pre-defined criteria like feature richness and availability, for an in-depth analysis on three publicly available corpora covering four entity types. Our results present a mixed picture and show that cross-corpus performance is significantly lower than the in-corpus performance. HunFlair2, the redesigned and extended successor of the HunFlair tool, showed the best performance on average, being closely followed by PubTator Central. Our results indicate that users of BTM tools should expect a lower performance than the original published one when applying tools in “the wild” and show that further research is necessary for more robust BTM tools. Availability and implementation All our models are integrated into the NLP framework flair: Https://github.com/flairNLP/flair. Code to reproduce our results is available at: Https://github.com/hu-ner/hunflair2-experiments. Supplementary information Supplementary information are available at Bioinformatics online.