Experiments on transfer learning architectures for biomedical relation extraction

Abstract

Relation extraction (RE) consists in identifying and structuring automatically relations of interest from texts. Recently, BERT improved the top performances for several NLP tasks, including RE. However, the best way to use BERT, within a machine learning architecture, and within a transfer learning strategy is still an open question since it is highly dependent on each specific task and domain. Here, we explore various BERT-based architectures and transfer learning strategies (i.e., frozen or fine-tuned) for the task of biomedical RE on two corpora. Among tested architectures and strategies, our *BERT-segMCNN with fine-tuning reaches performances higher than the state-of-the-art on the two corpora (1.73 % and 32.77 % absolute improvement on ChemProt and PGx-Corpus corpora respectively). More generally, our experiments illustrate the expected interest of fine-tuning with BERT, but also the unexplored advantage of using structural information (with sentence segmentation), in addition to the context classically leveraged by BERT.

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

“…BioBERT [121], BlueBERT [179], SciBERT [16], PubMedBERT [66], BioMegatron [215], Yuan et al [286], Wei et al [258], Hafiane et al [74], Thillaisundaram et al [234], Su et al [226], [225], Xue et al [272], Chen et al [34], Lin et al [135,136], Guan et al [68], Liu et al [143], Sui et al [227].…”

“…They investigated fine-tuned BERT on the clinical RE and combined additional BIO tag features. Similarly, Hafiane et al [74] investigated two transfer learning strategies: frozen and fine-tuning to adapt BERT on the biomedical RE. Thillaisundaram et al [234] adapted the SciBERT to the BioRE via fine-tuning the representation of the classification token (CLS).…”