DNAGPT: A Generalized Pre-trained Tool for Multiple DNA Sequence Analysis Tasks

Abstract

The success of the GPT series proves that GPT can extract general information from sequences, thereby benefiting all downstream tasks. This motivates us to use pre-trained models to explore the hidden information in DNA sequences. However, data and task requirements in DNA sequence analysis are complexity and diversity as DNA relevant data includes different types of information, such as sequences, expression levels, etc, while there is currently no model specifically designed for these characteristics. Hereby, we present DNAGPT, a generalized foundation model pre-trained on over 10 billion base pairs from 9 species which can be fine-tuned for any DNA sequence analysis task. Our model can simultaneously process or output DNA sequences and numbers. In addition, our unique token design allows users to design prompts according to their own task requirements, making it applicable to any type of task. We have evaluated our model on classification, regression, and generation tasks. We demonstrate that DNAGPT benefits from pre-training, and therefore can bring performance gains to any downstream task. Our model is not only a new attempt in the field of genomes analysis, but also provides a new direction for the application of foundation models in biology.

Evaluating the representational power of pre-trained DNA language models for regulatory genomics

“…Recently, there has been a surge of pre-trained gLMs [20][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35][36][37][38][38][39][40][41][42][43] . gLMs take as input DNA sequences that have undergone tokenization, an encoding scheme applied to either a single nucleotide or k-mer of nucleotides.…”

“…Current gLMs are composed of different choices for the tokenization, base architecture, language modeling objective, and pre-training data. Tokenization of DNA sequences is employed for either single nucleotide [20][21][22] or k-mer of fixed size [23][24][25] or a k-mer of variable sizes via byte-pair tokenization 26,27,45 , which aims to aggregate DNA in a manner that reduces the k-mer bias in the genome, a problem known as rare token imbalance. The base architecture is typically a stack of transformer layers 46 , with a vanilla multi-head self-attention [23][24][25][27][28][29][30][31] or an exotic attention variant (e.g., flash attention 26,47 , sparse attention 32,33 , or axial attention 34,48 ).…”

“…Tokenization of DNA sequences is employed for either single nucleotide [20][21][22] or k-mer of fixed size [23][24][25] or a k-mer of variable sizes via byte-pair tokenization 26,27,45 , which aims to aggregate DNA in a manner that reduces the k-mer bias in the genome, a problem known as rare token imbalance. The base architecture is typically a stack of transformer layers 46 , with a vanilla multi-head self-attention [23][24][25][27][28][29][30][31] or an exotic attention variant (e.g., flash attention 26,47 , sparse attention 32,33 , or axial attention 34,48 ). Alternatively, the base architecture has also been constructed with a stack of residual-connected convolution blocks, either with dilated convolutional layers 20 or implicit convolutions with a Hyena operator 21,22,49 .…”

“…Alternatively, the base architecture has also been constructed with a stack of residual-connected convolution blocks, either with dilated convolutional layers 20 or implicit convolutions with a Hyena operator 21,22,49 . The pre-training data can vary significantly, encompassing the whole genome of a single species 20,24,32 or the whole genomes across multiple species 23,25,26,28,33 or focused only within specific regions of the genomes, such as the untranslated regions (UTRs) 29 , pre-mRNA 30 , promoters 22 , coding regions [35][36][37] , non-coding RNA 40 , or conserved sites 34 .…”

Enhancing Recognition and Interpretation of Functional Phenotypic Sequences through Fine-Tuning Pre-Trained Genomic Models

“…However, their structural similarity to human language (long strings consisting of basic units such as bases or words) provides opportunities for modeling and interpreting DNA sequences using NLP methods[8, 9]. Scientists are increasingly leveraging pre-trained genomic models, leading to significant successes with Transformer-based frameworks [10, 11]and other language framework models[9]. For sequence classification evaluation, Researchers have constructed benchmark datasets for DNA classification and used modified CNN models as the baselines.…”