Decisive Roles of Sequence Distributions in the Generalizability ofde novoDeep Learning Models for RNA Secondary Structure Prediction

Abstract

The availability of sizeable RNA structure databases and powerful deep learning (DL) frameworks has prompted recent developments of DL models for RNA secondary structure prediction. Taking RNA sequences as the only inputs, the class of de novo DL models has demonstrated far superior performances than traditional algorithms. However, key questions remain over the statistical underpinning of such DL models which make no use of co-evolutionary information or physical laws of RNA folding. Here we present a quantitative study of the capacity and generalizability of a series of de novo DL models, with a minimal two-module architecture and no post-processing, under varied sequence distributions of seen and unseen datasets. Excellent performances are observed from our models with as few as 16K parameters, affirming their remarkable learning capacity. Our DL models prove to generalize well over non-identical unseen sequences, but the model generalizability degrades rapidly as the sequence distributions between the seen and unseen become dissimilar. Examinations of RNA family-specific behaviors manifest not only disparate family-dependent model performances but substantial generalization gaps within the same RNA family. We further determine how model generalization decreases with the decrease of sequence similarity via pairwise sequence alignment, providing quantitative insights into the limitations of statistical learning. Model generalizability thus poses major hurdles for practical uses of current single-sequence-based DL models and we discuss avenues for future advances of such de novo DL models for RNA secondary structure prediction.

Accurate prediction of RNA secondary structure including pseudoknots through solving minimum-cost flow with learned potentials

“…Similar results were observed on the TORNADO dataset (see Supplementary Table 9 for details). The underlying reasons may be the susceptibility to overfitting of deep learning models, and the low data coverage and density over diverse structures, as discussed in earlier studies 66,67 . To address this challenge, potential solutions include: 1) Actively exploring the integration of inductive bias terms, such as statistical energy terms, into the KnotFold model; 2) Considering the implementation of more ensemble strategies to introduce greater diversity to the models.…”

Deep learning models of RNA base-pairing structures generalize to unseen folds and make accurate zero-shot predictions of base-base interactions of RNA complexes

“…This raises concerns about the generalizability of deep learning models trained on such limited data. Previous studies by Szikszai et al 39 and Qiu 40 highlighted the challenges of deep-learning models when applied to unseen families not present in the training and validation sets. To evaluate the adaptability of SPOT-RNA and SPOT-RNA2 beyond their training and validation data, we conducted a test by removing all test set structures with the structural similarity score (TM-score) ≥ 0.3 compared to those in the training and validation sets.…”

Recent trends in RNA informatics: a review of machine learning and deep learning for RNA secondary structure prediction and RNA drug discovery

“…However, this also increases the risk of overfitting and poor prediction accuracy for structurally dissimilar sequences. The problem of overfitting is a prevalent issue not only in deep learning but also in other machine learning techniques with rich parametrization; it is particularly acute in deep learning as models can easily be scaled to an enormous number of parameters [ 75 , 78 ].…”