Cracking the black box of deep sequence-based protein-protein interaction prediction

Abstract

Identifying protein-protein interactions (PPIs) is crucial for deciphering biological pathways and their dysregulation. Numerous prediction methods have been developed as a cheap alternative to biological experiments, reporting phenomenal accuracy estimates. While most methods rely exclusively on sequence information, PPIs occur in 3D space. As predicting protein structure from sequence is an infamously complex problem, the almost perfect reported performances for PPI prediction seem dubious. We systematically investigated how much reproducible deep learning models depend on data leakage, sequence similarities, and node degree information and compared them to basic machine learning models. We found that overlaps between training and test sets resulting from random splitting lead to strongly overestimated performances, giving a false impression of the field. In this setting, models learn solely from sequence similarities and node degrees. When data leakage is avoided by minimizing sequence similarities between training and test, performances become random, leaving this research field wide open.

INTREPPPID - An Orthologue-Informed Quintuplet Network for Cross-Species Prediction of Protein-Protein Interaction

“…Studies by Dunham and Ganapathiraju in 2021 and Bernett et al . in 2023 corroborate Park and Marcotte’s 2012 findings (7, 8). In both studies, contemporary sequence-based PPI inference methods were tested on datasets that satisfy Park and Marcotte’s strictest validation schemes.…”

“…Furthermore, work by Hamp and Rost as well as Bernett et al . underscore an additional source of data leakage: distinct proteins with sequences which are nearly identical (8, 9). Amino acid sequences can be overwhelmingly redundant between distinct proteins.…”

Pitfalls of machine learning models for protein-protein interactions

“…On the other hand, sequence-based models have millions of parameters which give them the flexibility to recognise individual proteins and learn specific interaction patterns. Although this enables such models to make predictions without functional information, it also limits high performance to proteins present in the training set, and make them particularly susceptible to data leakage [17]. This likely explains the poor results of sequence models on previously unseen proteins and cross-species datasets, something also observed by Dunham et al in their benchmarking effort [16].…”

“…Yet, despite a wealth of tools, the mechanics and consequences of the underlying inference are still poorly understood, and it is unclear why models with similar performance make vastly different predictions. Reported performance scores often cannot be compared or replicated due to proprietary data and inconsistent or flawed assessment methods [16], [17]. This prompted recent efforts to benchmark published PPI prediction models more rigorously using common datasets and testing strategies [16], [18].…”

Rapid protein-protein interaction network creation from multiple sequence alignments with Deep Learning

“…At the same time, single-sequence methods (i.e. those not dependent on an MSA) for PPI prediction have been found to be unreliable [7]. In contrast to other methods, AF for single chains has not been trained on PPIs and, therefore, successful PPI prediction indicates that the network generalizes to this new task.…”