A Self-Attention Model for Inferring Cooperativity between Regulatory Features

Abstract

Motivation:Deep learning has demonstrated its predictive power in modeling complex biological phenomena such as gene expression. The value of these models hinges not only on their accuracy, but also on the ability to extract biologically relevant information from the trained models. While there has been much recent work on developing feature attribution methods that provide the most important features for a given sequence, inferring cooperativity between regulatory elements, which is the hallmark of phenomena such as gene expression, remains an open problem. Results: We present SATORI, a Self-ATtentiOn based model to predict Regulatory element Interactions. Our approach combines convolutional and recurrent layers with a self-attention mechanism that helps us capture a global view of the landscape of interactions between regulatory elements in a sequence. We evaluate our method on simulated data and three complex datasets: human TAL1-GATA1 transcription factor ChIP-Seq, DNase I Hypersensitive Sites (DHSs) in human promoters across 164 cell lines, and genome-wide DNase I-Seq and ATAC-Seq peaks across 36 arabidopsis samples. In each of the three experiments SATORI identified numerous statistically significant TF interactions, many of which have been previously reported. Our approach not only provides a global, biologically relevant set of interactions but, unlike existing methods, it does not require a computationally expensive postprocessing step. Availability: The source code for SATORI is available at https://github.com/fahadahaf/SATORI.

Learning the Regulatory Code of Gene Expression

“…saliency maps ( Simonyan et al, 2013 ) and 4) higher-order interactions among sequence elements, which can be assessed e.g. by using association rule analysis ( Naulaerts et al, 2015 ; Zrimec et al, 2020 ), second-order perturbations ( Koo et al, 2018 ), self-attention networks ( Ullah and Ben-Hur, 2020 ) or by visualizing kernels in deeper layers ( Maslova et al, 2020 ) [interested readers are referred to ( Eraslan et al, 2019a ; Koo and Ploenzke, 2020a )]. Moreover, attention mechanisms were recently shown to be more effective in discovering known TF-binding motifs compared to non-attentive DNNs ( Park et al, 2020 ), as the learned attention weights correlate with informative inputs, such as DNase-Seq coverage and DNA motifs ( Chen et al, 2021 ), and they can provide better interpretation than other established feature visualization methods, such as saliency maps ( Lanchantin et al, 2016 ; Singh et al, 2017 ).…”

Characterizing collaborative transcription regulation with a graph-based deep learning approach

“…Whereas ECHO and ChromeGCN [13] explicitly leverage chromatin contacts, DNA interactions are implicitly captured by SATORI [12] and Basenji [11]. SATORI captures TF-TF interactions by combining CNN with self-attention mechanisms.…”

“…According to whether 3D chromatin organization is utilized, we categorize the deep learning based computational works for predicting chromatin features into sequence-based and graph-based models. Well-known sequence-based models, such as DeepSEA [7], DanQ [8], DeepBind [9], Basset [10], Basenji [11], and SATORI [12], predict chromatin features only from DNA sequences and ignore the informative chromatin structures. To the best of our knowledge, the only graph-based chromatin feature prediction model ChromeGCN [13] uses a gated graph convolution network to leverage the neighborhood information from a 1kb resolution Hi-C contact map, but it does not fully characterize cooperation among chromatin features.…”

Designing Interpretable Convolution-Based Hybrid Networks for Genomics

“…However, extracting these interactions is not straightforward. Alternatively, MHA, a key component of transformers, can provide an "interpretable" attention map to reveal learned activity between pairs of convolutional filters (Ullah & Ben-Hur, 2021). However, such an attention map is only intrinsically interpretable if the convolutional layer(s) learn robust motif representations and are identifiable in the attention maps.…”

“…Recently, several hybrid networks that build upon convolutional layers with architectures developed for natural language processing, including bidirectional long-short-term memory (BiLSTM) (Quang & Xie, 2016;Minnoye et al, 2020), multi-head attention (MHA) (Li et al, 2020;Ullah & Ben-Hur, 2021), and transformer encoders (Ji et al, 2020;Avsec et al, 2021a), have demonstrated improved performance relative to pure CNNs. BiLSTMs can, in principle, capture long-range motif interactions.…”