DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome

Abstract

Motivation Deciphering the language of non-coding DNA is one of the fundamental problems in genome research. Gene regulatory code is highly complex due to the existence of polysemy and distant semantic relationship, which previous informatics methods often fail to capture especially in data-scarce scenarios. Results To address this challenge, we developed a novel pre-trained bidirectional encoder represen-tation, named DNABERT, to capture global and transferrable understanding of genomic DNA sequences based on up and downstream nucleotide contexts. We compared DNABERT to the most widely used programs for genome-wide regulatory elements prediction and demonstrate its ease of use, accuracy, and efficiency. We show that the single pre-trained transformers model can simultaneously achieve state-of-the-art performance on prediction of promoters, splice sites, and transcription factor binding sites, after easy fine-tuning using small task-specific labelled data. Further, DNABERT enables direct visualization of nucleotide-level importance and semantic relationship within input sequences for better interpretability and accurate identification of conserved sequence motifs and functional genetic variant candidates. Finally, we demonstrate that pre-trained DNABERT with human genome can even be readily applied to other organisms with exceptional performance. We anticipate that the pre-trained DNABERT model can be fined tuned to many other sequence analyses tasks. Availability The source code, pretrained and finetuned model for DNABERT are available at GitHub https://github.com/jerryji1993/DNABERT Supplementary information Supplementary data are available at Bioinformatics online.

Effective gene expression prediction from sequence by integrating long-range interactions

“…1). Transformers are a class of deep learning models that have achieved substantial breakthroughs in natural language processing (NLP) 6,7 and were also recently applied to model short DNA sequences 8 . They consist of attention layers that transform each position in the input sequence by computing a weighted sum across the representations of all other positions in the sequence.…”

BindVAE: Dirichlet variational autoencoders for de novo motif discovery from accessible chromatin

“…Supervised deep learning methods for the prediction of TF occupancy data and and chromatin accessibility are numerous, ranging from the early deep convolutional neural network based pipelines such as DeepSEA 10 and Basset 9 to more recent approaches usually mirroring advances in deep learning methods for natural language processing, such as the LSTM-based DanQ 23 , Basenji using dilated CNNs 11 , DeepSite 24 , and DNA-BERT 25 . While these models have produced highly accurate predictions of TF occupancy, the interpretation of models requires detailed feature attribution (e.g.…”

Representation learning applications in biological sequence analysis

“…DNABERT , in contrast, is the only model, currently, to pre-train BERT-based models using a whole human reference genome [68]. During preprocessing, the genome, whose gaps and unannotated regions were excluded, was split into 5 to 510 consequent nucleotide sequences without overlapping and subsequently converted to 3-to 6-mer representations.…”