Prediction of RNA-protein interactions using a nucleotide language model

Yamada, Keisuke; Hamada, Michiaki

doi:10.1101/2021.04.27.441365

Cited by 6 publications

(5 citation statements)

References 73 publications

(78 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Wei et al [38] have provided a review of the use of deep learning in RNAprotein interaction prediction. Yamada et al [39] have developed a method to accurately identify RNA sequences that interact with a particular protein by using the DNABERT model [40] that is pre-trained using the human genome. Although our method does not use deep learning, we expect to achieve higher accuracy in prediction by using a pretrained BERT model, which could be improved through the application of deep learning relatively easily.…”

Section: Discussionmentioning

confidence: 99%

A Max-Margin Model for Predicting Residue—Base Contacts in Protein–RNA Interactions

Kashiwagi

Sato

Sakakibara

2021

Life

View full text Add to dashboard Cite

Protein–RNA interactions (PRIs) are essential for many biological processes, so understanding aspects of the sequences and structures involved in PRIs is important for unraveling such processes. Because of the expensive and time-consuming techniques required for experimental determination of complex protein–RNA structures, various computational methods have been developed to predict PRIs. However, most of these methods focus on predicting only RNA-binding regions in proteins or only protein-binding motifs in RNA. Methods for predicting entire residue–base contacts in PRIs have not yet achieved sufficient accuracy. Furthermore, some of these methods require the identification of 3D structures or homologous sequences, which are not available for all protein and RNA sequences. Here, we propose a prediction method for predicting residue–base contacts between proteins and RNAs using only sequence information and structural information predicted from sequences. The method can be applied to any protein–RNA pair, even when rich information such as its 3D structure, is not available. In this method, residue–base contact prediction is formalized as an integer programming problem. We predict a residue–base contact map that maximizes a scoring function based on sequence-based features such as k-mers of sequences and the predicted secondary structure. The scoring function is trained using a max-margin framework from known PRIs with 3D structures. To verify our method, we conducted several computational experiments. The results suggest that our method, which is based on only sequence information, is comparable with RNA-binding residue prediction methods based on known binding data.

show abstract

Section: Discussionmentioning

confidence: 99%

A Max-Margin Model for Predicting Residue—Base Contacts in Protein–RNA Interactions

Kashiwagi

Sato

Sakakibara

2021

Life

View full text Add to dashboard Cite

show abstract

“…The i th self-attention head is computed as where Attention h = { a ij } is a scoring matrix, in which a ij denotes the attention weight that the Query token t i gets from then Key token t j . This matrix is widely used for representing and exploring the binding between tokens (33, 49, 56).…”

Section: Methodsmentioning

confidence: 99%

MuLan-Methyl - Multiple Transformer-based Language Models for Accurate DNA Methylation Prediction

Zeng

Gautam

Huson

2023

Preprint

View full text Add to dashboard Cite

Transformer-based language models are successfully used to address massive text-related tasks. DNA methylation is an important epigenetic mechanism and its analysis provides valuable insights into gene regulation and biomarker identification. Several deep learning-based methods have been proposed to identify DNA methylation and each seeks to strike a balance between computational effort and accuracy. Here, we introduce a MuLan-Methyl, a deep learning framework for predicting DNA methylation sites, which is based on multiple (five) popular transformer-based language models. The framework identifies methylation sites for three different types of DNA methylation, namely N6-adenine (6mA), N4-cytosine (4mC), and 5-hydroxymethylcytosine (5hmC). Each of the five employed language models is adapted to the task using the "pre-train and fine-tune" paradigm. Pre-training is performed on a custom corpus consisting of DNA fragments and taxonomy lineages using self-supervised learning. Fine-tuning then aims at predicting the DNA-methylation status of each type. The five models are used to collectively predict the DNA methylation status. We report excellent performance of MuLan-Methyl on a benchmark dataset. Moreover, we show that the model captures characteristic differences between different species that are relevant for methylation. This work demonstrates that language models can be successfully adapted to this domain of application and that joint utilization of different language models improves model performance.

show abstract

“…Natural language inference BioELMo [93], BlueBERT [179], Sharma et al [212], He et al [78], Zhu et al [302]. Protein and DNA sequence [9], [80], [197], [202], MSA Transformer [198], ProtTrans [54], ProGen [146], DNABERT [90], [275] feasible to make use of unlabelled protein data. In detail, Alphafold2 adopts an auxiliary BERT-like loss to predict pre-masked residues in multiple sequence alignments (MSAs).…”

Section: Question Answeringmentioning

confidence: 99%

“…Yamada and Hamada [275] pre-trains a BERT on RNA sequences and RNA-binding protein sequences. All the LMs remain largely the same as those used for human language data.…”

Section: Question Answeringmentioning

confidence: 99%

“…Hong et al [85] proposed to pre-train DNA vectors to encode enhancers and promoters, and then Incorporated the attention mechanism to predicting long-range enhancer-promoter interactions (EPIs). Yamada et al [276] proposed a novel method based on the BERT to predict the interactions between RNA sequences and RNA-binding proteins (RBPs), in which the BERT model is pre-trained on the human reference genome.…”

Section: Dna Related Applicationsmentioning

confidence: 99%

See 1 more Smart Citation

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Wang¹,

Xie²,

Pei³

et al. 2021

Preprint

View full text Add to dashboard Cite

Pre-trained language models (PLMs) have been the de facto paradigm for most natural language processing (NLP) tasks. This also benefits biomedical domain: researchers from informatics, medicine, and computer science (CS) communities proposes various PLMs trained on biomedical datasets, e.g., biomedical text, electronic health records, protein, and DNA sequences for various biomedical tasks. However, the cross-discipline characteristics of biomedical PLMs hinder their spreading among communities; some existing works are isolated from each other without comprehensive comparison and discussions. It expects a survey that not only systematically reviews recent advances of biomedical PLMs and their applications but also standardizes terminology and benchmarks. In this paper, we summarize the recent progress of pre-trained language models in the biomedical domain and their applications in biomedical downstream tasks. Particularly, we discuss the motivations and propose a taxonomy of existing biomedical PLMs. Their applications in biomedical downstream tasks are exhaustively discussed. At last, we illustrate various limitations and future trends, which we hope can provide inspiration for the future research of the research community.CCS Concepts: • Computing methodologies → Natural language processing; Natural language generation; Neural networks; Bio-inspired approaches.

show abstract

Prediction of RNA-protein interactions using a nucleotide language model

Cited by 6 publications

References 73 publications

A Max-Margin Model for Predicting Residue—Base Contacts in Protein–RNA Interactions

A Max-Margin Model for Predicting Residue—Base Contacts in Protein–RNA Interactions

MuLan-Methyl - Multiple Transformer-based Language Models for Accurate DNA Methylation Prediction

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Contact Info

Product

Resources

About