Sequence similarity governs generalizability of de novo deep learning models for RNA secondary structure prediction

Abstract

Making no use of physical laws or co-evolutionary information, de novo deep learning (DL) models for RNA secondary structure prediction have achieved far superior performances than traditional algorithms. However, their statistical underpinning raises the crucial question of generalizability. We present a quantitative study of the performance and generalizability of a series of de novo DL models, with a minimal two-module architecture and no post-processing, under varied similarities between seen and unseen sequences. Our models demonstrate excellent expressive capacities and outperform existing methods on common benchmark datasets. However, model generalizability, i.e., the performance gap between the seen and unseen sets, degrades rapidly as the sequence similarity decreases. The same trends are observed from several recent DL and machine learning models. And an inverse correlation between performance and generalizability is revealed collectively across all learning-based models with wide-ranging architectures and sizes. We further quantitate how generalizability depends on sequence and structure identity scores via pairwise alignment, providing unique quantitative insights into the limitations of statistical learning. Generalizability thus poses a major hurdle for deploying de novo DL models in practice and various pathways for future advances are discussed.

RNA3DB: A structurally-dissimilar dataset split for training and benchmarking deep learning models for RNA structure prediction

“…The sparsity of data alone could be responsible for the poor performance of deep learning methods dependent on millions of parameters to predict RNA structure, as it was seen at CASP15 [21, 22] for 3D structure prediction, as well as for 2D structure prediction [14, 15, 16, 17]. Other factors possibly handicapping the prediction of RNA structure are the more complex RNA backbone geometry that involves more atoms and degrees of freedom, as well as global nature of RNA secondary structure.…”

“…The structural separation is needed in order to distinguish whether the methods are able to generalize to new structures or not; otherwise data leakage results in overfitting [18]. Recent studies of deep learning methods have also shown the need to separate structures between training and testing sets [14, 15, 16, 17, 23]. The goal of RNA3DB is to provide such structurally dissimilar grouping and splitting of the existing RNA 3D structures present in PDB.…”

“…Despite this, most RNA biologists do not consider deep learning methods to be state-of-the-art for RNA structure prediction [13] for secondary or tertiary structures. Publications began pointing out issues with generalization to unseen sequences in deep learning methods for secondary structure prediction in 2022 [14, 15, 16, 17], although this overfitting behavior was already well-known for probabilistic and Markov random field models trained on structural RNA data since 2012 [18]. With the increased interest in RNA, partly attributed to AlphaFold’s success, and partly due to the rise of RNA-based therapeutics [19], CASP15 joined RNA-Puzzles [20] and added 12 RNA-only targets to the competition [21, 22], where all deep learning methods performed comparatively poorly to traditional tools [23].…”

Accurate prediction of RNA secondary structure including pseudoknots through solving minimum-cost flow with learned potentials

“…Similar results were observed on the TORNADO dataset (see Supplementary Table 9 for details). The underlying reasons may be the susceptibility to overfitting of deep learning models, and the low data coverage and density over diverse structures, as discussed in earlier studies 66 , 67 . To address this challenge, potential solutions include: 1) Actively exploring the integration of inductive bias terms, such as statistical energy terms, into the KnotFold model; 2) Considering the implementation of more ensemble strategies to introduce greater diversity to the models.…”

Deep learning models of RNA base-pairing structures generalize to unseen folds and make accurate zero-shot predictions of base-base interactions of RNA complexes

“…This raises concerns about the generalizability of deep learning models trained on such limited data. Previous studies by Szikszai et al 39 and Qiu 40 highlighted the challenges of deep-learning models when applied to unseen families not present in the training and validation sets. To evaluate the adaptability of SPOT-RNA and SPOT-RNA2 beyond their training and validation data, we conducted a test by removing all test set structures with the structural similarity score (TM-score) ≥ 0.3 compared to those in the training and validation sets.…”