Novel transformer networks for improved sequence labeling in genomics

Abstract

In genomics, a wide range of machine learning methods is used to annotate biological sequences w.r.t. interesting positions such as transcription start sites, translation initiation sites, methylation sites, splice sites, promotor start sites, etc. In recent years, this area has been dominated by convolutional neural networks, which typically outperform older methods as a result of automated scanning for influential sequence motifs. As an alternative, we introduce in this paper transformer architectures for whole-genome sequence labeling tasks. We show that those architectures, which have been recently introduced for natural language processing, allow for a fast processing of long DNA sequences. We optimize existing networks and define a new way to calculate attention, resulting in state-of-the-art performances. To demonstrate this, we evaluate our transformer model architecture on several sequence labeling tasks, and find it to outperform specialized models for the annotation of transcription start sites, translation initiation sites and 4mC methylation in E. coli. In addition, the use of the full genome for model training and evaluation results in unbiased performance metrics, facilitating future benchmarking.

Explainable Transformer Models for Functional Genomics in Prokaryotes

“…The performance of the model is in line with state-of-the-art performances reported in our previous work [6]. The evaluation of the attention weights of the transformer network revealed a notable link to the promoter regions of importance for the RNAP binding process.…”

“…We recently introduced a custom transformer architecture for processing the full genomic sequence and obtained state-of-the-art performances for the annotation of TSSs, translation initiation sites and methylation sites [6]. Importantly, the transformer architecture does not assert the relative positions of the input nucleotides with respect to the output label.…”

“…a TSS) and those that do not. This presents an extra challenge for predictive tools that is a likely explanation towards the lower predictive performance of TSSs in comparison with translation initiation sites, where a ROC AUC of 0.998 is obtained [6]. With respect to RNA polymerase interference, post-processing of the model output profiles for both strands could be viable for further improvement of the predictive model.…”

“…This relevance is calculated at each node and is reflected by weighing of each of the inputs hidden states of the previous layer [34]. A more in-depth explanation of the model architecture is given in Supplementary File 1 and in the article where the transformer model for genome annotations is first described [6]. Through hyperparameter tuning, a model architecture is selected (l: 512, k:6).…”

“…In 2017, Umarov et al further studied the use of convolutional neural networks for the annotation of TSSs [33]. More recently, we have introduced a transformer-based model for genome annotation tasks, and have achieved state-of-the-art results for the annotation of TSSs, translation initiation sites and methylation sites [6]. The introduced architecture is adapted from the transformer-XL [7], first introduced in the field of natural language processing, and is well suited to process the long nucleotide sequence data.…”