STonKGs: a sophisticated transformer trained on biomedical text and knowledge graphs

Abstract

Motivation The majority of biomedical knowledge is stored in structured databases or as unstructured text in scientific publications. This vast amount of information has led to numerous machine learning-based biological applications using either text through natural language processing (NLP) or structured data through knowledge graph embedding models (KGEMs). However, representations based on a single modality are inherently limited. Results To generate better representations of biological knowledge, we propose STonKGs, a Sophisticated Transformer trained on biomedical text and Knowledge Graphs (KGs). This multimodal Transformer uses combined input sequences of structured information from KGs and unstructured text data from biomedical literature to learn joint representations in a shared embedding space. First, we pre-trained STonKGs on a knowledge base assembled by the Integrated Network and Dynamical Reasoning Assembler (INDRA) consisting of millions of text-triple pairs extracted from biomedical literature by multiple NLP systems. Then, we benchmarked STonKGs against three baseline models trained on either one of the modalities (i.e., text or KG) across eight different classification tasks, each corresponding to a different biological application. Our results demonstrate that STonKGs outperforms both baselines, especially on the more challenging tasks with respect to the number of classes, improving upon the F1-score of the best baseline by up to 0.084 (i.e., from 0.881 to 0.965). Finally, our pre-trained model as well as the model architecture can be adapted to various other transfer learning applications. Availability We make the source code and the Python package of STonKGs available at GitHub (https://github.com/stonkgs/stonkgs) and PyPI (https://pypi.org/project/stonkgs/). The pre-trained STonKGs models and the task-specific classification models are respectively available at https://huggingface.co/stonkgs/stonkgs-150k and https://zenodo.org/communities/stonkgs. Supplementary information Supplementary data are available at Bioinformatics online.

A Bibliometric Analysis of Recent Developments and Trends in Knowledge Graph Research (2013–2022)

“…At the same time, in terms of published articles, Lehmann ranked the top in the ranking list with 17 articles, and he mainly concentrated on the analysis of semantic question-answering systems and link prediction of knowledge graphs [26], [27]. Fernández took second place with 12 publications, which mainly specialized in the application of knowledge graph embedding models and natural language processing in the domain of biomedical literature [28]. What's more, Tan and the other two authors tied for third place with 11 articles, and he investigated generic knowledge graph embedding models and knowledge graph representation because his most contributed topics were domain relation extraction and relational knowledge prediction [29], [30].…”

Transformers and LLMs as the New Benchmark in Early Cancer Detection

“…Nerella et al provided a thorough survey on Transformers in healthcare, highlighting their versatility across fields like genomics and patient care [4]. Balabin introduced Multimodal Transformers, showcasing their capability to manage complex biomedical data [5]. Thirunavukarasu et al discussed LLMs in medicine, emphasizing the necessity for models that comprehend complex medical terminology [6].…”

CoVEffect: interactive system for mining the effects of SARS-CoV-2 mutations and variants based on deep learning

“…CoVEffect stems from this thread of works, but it is carefully adapted to solve a more complex task: that of predicting a series of tuples from SARS-CoV-2–related abstracts where we consider a variation, its effect, and the change of its level. Each of the currently available systems supports only one user-driven annotation [ 47 ], predictions of single independent annotations with ontological terms [ 48 ], or biomedical general-purpose triplets based on existing knowledge graphs [ 49 ], especially targeted to protein–protein interactions [ 50 ]. These correspond to different tasks than the one performed by CoVEffect, and the described approaches do not allow for online modifications of the training dataset.…”