Assessment of community efforts to advance computational prediction of protein-protein interactions

Abstract

Comprehensive insights from the human protein-protein interaction (PPI) network, known as the human interactome, can provide important insights into the molecular mechanisms of complex biological processes and diseases. Despite the remarkable experimental efforts undertaken to date to determine the structure of the human interactome, many PPIs remain unmapped. Computational approaches, especially network-based methods, can facilitate the identification of new PPIs. Many such approaches have been proposed. However, a systematic evaluation of existing network-based methods in predicting PPIs is still lacking. Here, we report community efforts initiated by the International Network Medicine Consortium to benchmark the ability of 24 representative network-based methods to predict PPIs across five different interactomes, including a synthetic interactome generated by the duplication-mutation-complementation model, and the interactomes of four different organisms: A. thaliana, C. elegans, S. cerevisiae, and H. sapiens. We selected the top-seven methods through a computational validation on the human interactome. We next experimentally validated their top-500 predicted PPIs (in total 3,276 predicted PPIs) using the yeast two-hybrid assay, finding 1,177 new human PPIs (involving 633 proteins). Our results indicate that task-tailored similarity-based methods, which leverage the underlying network characteristics of PPIs, show superior performance over other general link prediction methods. Through experimental validation, we confirmed that the top-ranking methods show promising performance externally. For example, from the top 500 PPIs predicted by an advanced similarity-base method [MPS(B&T)], 430 were successfully tested by Y2H with 376 testing positive, yielding a precision of 87.4%. These results establish advanced similarity-based methods as powerful tools for the prediction of human PPIs.

Computational Inference of Synaptic Polarities in Neuronal Networks

“…At the same threshold SPA unveils an additional 63 synaptic polarities, altogether identifying 112 polarities at 95% precision, as illustrated in Figure 2F. Our findings indicate that signed extensions of link prediction methods, [28] as well as the growing body of network sign prediction methods [29][30][31][32] should be further explored.…”

Pitfalls of machine learning models for protein-protein interactions

“…By being freely accessible and using standard identifiers for proteins, B4PPI can be used by any researcher working on in silico PPI prediction to investigate the inference mechanisms of their models. Contrary to previous benchmarking efforts [18], which it can complement, B4PPI leverages large professionally curated databases. An example reporting sheet is presented that includes relevant metrics, from PR and ROC curves to runtime and carbon footprint, to ensure the models released can be trusted and encourage wider use of PPI imputation for downstream analysis.…”

“…This is addressed by the PR curve where precision puts an emphasis on positive examples. It has also been shown that ROC tend to overestimate the performance of PPI prediction tools [18], but most published models still report it, which justifies the inclusion of both metrics.…”

“…Reported performance scores often cannot be compared or replicated due to proprietary data and inconsistent or flawed assessment methods [16], [17]. This prompted recent efforts to benchmark published PPI prediction models more rigorously using common datasets and testing strategies [16], [18]. These benchmarking studies hinted at some fundamental differences in how algorithms predict PPIs, e.g.…”

State of the Interactomes: an evaluation of molecular networks for generating biological insights

“…Finally, we leveraged advances in edge prediction algorithms 19 and AlphaFold-based modeling 20 to evaluate the interaction prediction performance of the panel of interactomes. Using 10-fold crossvalidation with L3 12 (paths of length 3) and MPS(T) 19,47 (Maximum similarity, Preferential attachment Score) algorithms, we predicted interactions for each interactome, excluding interactomes with greater than 1.5M interactions due to computational constraints. The predicted interactions were assessed against held-out and gold-standard interactions.…”