Benchmark and Best Practices for Biomedical Knowledge Graph Embeddings

Balažević, Ivana; Allen, Carl; Chawla, Daniel G.; Brandt, Cynthia; Taylor, Richard A.

doi:10.18653/v1/2020.bionlp-1.18

Cited by 21 publications

(20 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our results show RotatE is often the best performing model of the five on both datasets and throughout all the experiments. This reinforces previous similar findings [11,2] and shows RotatE to be a strong baseline in the context of drug discovery. Our results also highlight that older approaches like TransE can still be very competitive given an optimised training and hyperparameter setup.…”

Section: Discussionsupporting

confidence: 92%

“…The performance of five knowledge graph embedding approaches (TransE, ComplEx, DistMult, SimplE and RotatE) have been compared on a knowledge graph constructed from the SNOMED resource [11]. The models are assessed on the tasks of link prediction, visualisation and entity classification, with a limited grid-search being performed to choose the hyperparameters.…”

Section: Biomedical Domain Specific Evaluationsmentioning

confidence: 99%

See 1 more Smart Citation

Understanding the Performance of Knowledge Graph Embeddings in Drug Discovery

Bonner,

Barrett,

et al. 2021

Preprint

View full text Add to dashboard Cite

Knowledge Graphs (KG) and associated Knowledge Graph Embedding (KGE) models have recently begun to be explored in the context of drug discovery and have the potential to assist in key challenges such as target identification. In the drug discovery domain, KGs can be employed as part of a process which can result in lab-based experiments being performed, or impact on other decisions, incurring significant time and financial costs and most importantly, ultimately influencing patient healthcare. For KGE models to have impact in this domain, a better understanding of not only of performance, but also the various factors which determine it, is required. In this study we investigate, over the course of many thousands of experiments, the predictive performance of five KGE models on two public drug discoveryoriented KGs. Our goal is not to focus on the best overall model or configuration, instead we take a deeper look at how performance can be affected by changes in the training setup, choice of hyperparameters, model parameter initialisation seed and different splits of the datasets. Our results highlight that these factors have significant impact on performance and can even affect the ranking of models. Indeed these factors should be reported along with model architectures to ensure complete reproducibility and fair comparisons of future work, and we argue this is critical for the acceptance of use, and impact of KGEs in a biomedical setting. To aid reproducibility of our own work, we release all experimentation code.Preprint. Under review.

show abstract

Section: Discussionsupporting

confidence: 92%

Section: Biomedical Domain Specific Evaluationsmentioning

confidence: 99%

Understanding the Performance of Knowledge Graph Embeddings in Drug Discovery

Bonner,

Barrett,

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The BERT model is a neural network transformer‐based model that generates embeddings for sentences to capture meaning. We used Clinical BERT, a variant of BERT fine‐tuned on biomedical and clinical text corpora (MIMIC‐III and PubMed) 14 …”

Section: Methodsmentioning

confidence: 99%

Early identification of patients with acute gastrointestinal bleeding using natural language processing and decision rules

Shung

Tsay

Laine

et al. 2021

J of Gastro and Hepatol

Self Cite

View full text Add to dashboard Cite

Background and Aim Guidelines recommend risk stratification scores in patients presenting with gastrointestinal bleeding (GIB), but such scores are uncommonly employed in practice. Automation and deployment of risk stratification scores in real time within electronic health records (EHRs) would overcome a major impediment. This requires an automated mechanism to accurately identify (“phenotype”) patients with GIB at the time of presentation. The goal is to identify patients with acute GIB by developing and evaluating EHR‐based phenotyping algorithms for emergency department (ED) patients. Methods We specified criteria using structured data elements to create rules for identifying patients and also developed multiple natural language processing (NLP)‐based approaches for automated phenotyping of patients, tested them with tenfold cross‐validation for 10 iterations (n = 7144) and external validation (n = 2988) and compared them with a standard method to identify patient conditions, the Systematized Nomenclature of Medicine. The gold standard for GIB diagnosis was the independent dual manual review of medical records. The primary outcome was the positive predictive value. Results A decision rule using GIB‐specific terms from ED triage and ED review‐of‐systems assessment performed better than the Systematized Nomenclature of Medicine on internal validation and external validation (positive predictive value = 85% confidence interval:83%–87% vs 69% confidence interval:66%–72%; P < 0.001). The syntax‐based NLP algorithm and Bidirectional Encoder Representation from Transformers neural network‐based NLP algorithm had similar performance to the structured‐data fields decision rule. Conclusions An automated decision rule employing GIB‐specific triage and review‐of‐systems terms can be used to trigger EHR‐based deployment of risk stratification models to guide clinical decision making in real time for patients with acute GIB presenting to the ED.

show abstract

“…Essentially, a KGE model maps entities and relations to embedding spaces by using a predefined scoring function. Due to their growing popularity and the availability of implementation methods, KGEs have recently been applied to various domains, including biomedical knowledge graphs [ 26 ]. Chang et al [ 26 ] showed that using KGEs for learning concept embeddings from medical terminologies and knowledge graphs is arguably a more principled and effective approach than using previous methods based on skip-gram–based models like Cui2Vec [ 27 ] or network embedding–based models like Snomed2Vec [ 28 ].…”

Section: Methodsmentioning

confidence: 99%

“…Due to their growing popularity and the availability of implementation methods, KGEs have recently been applied to various domains, including biomedical knowledge graphs [ 26 ]. Chang et al [ 26 ] showed that using KGEs for learning concept embeddings from medical terminologies and knowledge graphs is arguably a more principled and effective approach than using previous methods based on skip-gram–based models like Cui2Vec [ 27 ] or network embedding–based models like Snomed2Vec [ 28 ]. Although we initially used Cui2Vec for our entity vectors at the time of submission, we later used SNOMED CT KGEs after they became available in recent months.…”

Section: Methodsmentioning

confidence: 99%

Incorporating Domain Knowledge Into Language Models by Using Graph Convolutional Networks for Assessing Semantic Textual Similarity: Model Development and Performance Comparison

Lin¹,

Brandt²,

Taylor³

2021

JMIR Med Inform

Self Cite

View full text Add to dashboard Cite

Background Although electronic health record systems have facilitated clinical documentation in health care, they have also introduced new challenges, such as the proliferation of redundant information through the use of copy and paste commands or templates. One approach to trimming down bloated clinical documentation and improving clinical summarization is to identify highly similar text snippets with the goal of removing such text. Objective We developed a natural language processing system for the task of assessing clinical semantic textual similarity. The system assigns scores to pairs of clinical text snippets based on their clinical semantic similarity. Methods We leveraged recent advances in natural language processing and graph representation learning to create a model that combines linguistic and domain knowledge information from the MedSTS data set to assess clinical semantic textual similarity. We used bidirectional encoder representation from transformers (BERT)–based models as text encoders for the sentence pairs in the data set and graph convolutional networks (GCNs) as graph encoders for corresponding concept graphs that were constructed based on the sentences. We also explored techniques, including data augmentation, ensembling, and knowledge distillation, to improve the model’s performance, as measured by the Pearson correlation coefficient (r). Results Fine-tuning the BERT_base and ClinicalBERT models on the MedSTS data set provided a strong baseline (Pearson correlation coefficients: 0.842 and 0.848, respectively) compared to those of the previous year’s submissions. Our data augmentation techniques yielded moderate gains in performance, and adding a GCN-based graph encoder to incorporate the concept graphs also boosted performance, especially when the node features were initialized with pretrained knowledge graph embeddings of the concepts (r=0.868). As expected, ensembling improved performance, and performing multisource ensembling by using different language model variants, conducting knowledge distillation with the multisource ensemble model, and taking a final ensemble of the distilled models further improved the system’s performance (Pearson correlation coefficients: 0.875, 0.878, and 0.882, respectively). Conclusions This study presents a system for the MedSTS clinical semantic textual similarity benchmark task, which was created by combining BERT-based text encoders and GCN-based graph encoders in order to incorporate domain knowledge into the natural language processing pipeline. We also experimented with other techniques involving data augmentation, pretrained concept embeddings, ensembling, and knowledge distillation to further increase our system’s performance. Although the task and its benchmark data set are in the early stages of development, this study, as well as the results of the competition, demonstrates the potential of modern language model–based systems to detect redundant information in clinical notes.

show abstract

Benchmark and Best Practices for Biomedical Knowledge Graph Embeddings

Cited by 21 publications

References 17 publications

Understanding the Performance of Knowledge Graph Embeddings in Drug Discovery

Understanding the Performance of Knowledge Graph Embeddings in Drug Discovery

Early identification of patients with acute gastrointestinal bleeding using natural language processing and decision rules

Incorporating Domain Knowledge Into Language Models by Using Graph Convolutional Networks for Assessing Semantic Textual Similarity: Model Development and Performance Comparison

Contact Info

Product

Resources

About