Ten quick tips for sequence-based prediction of protein properties using machine learning

Abstract

The ubiquitous availability of genome sequencing data explains the popularity of machine learning-based methods for the prediction of protein properties from their amino acid sequences. Over the years, while revising our own work, reading submitted manuscripts as well as published papers, we have noticed several recurring issues, which make some reported findings hard to understand and replicate. We suspect this may be due to biologists being unfamiliar with machine learning methodology, or conversely, machine learning experts may miss some of the knowledge needed to correctly apply their methods to proteins. Here, we aim to bridge this gap for developers of such methods. The most striking issues are linked to a lack of clarity: how were annotations of interest obtained; which benchmark metrics were used; how are positives and negatives defined. Others relate to a lack of rigor: If you sneak in structural information, your method is not sequence-based; if you compare your own model to “state-of-the-art,” take the best methods; if you want to conclude that some method is better than another, obtain a significance estimate to support this claim. These, and other issues, we will cover in detail. These points may have seemed obvious to the authors during writing; however, they are not always clear-cut to the readers. We also expect many of these tips to hold for other machine learning-based applications in biology. Therefore, many computational biologists who develop methods in this particular subject will benefit from a concise overview of what to avoid and what to do instead.

Transcription factor prediction using protein 3D secondary structures

“…To see how the performances of each of StrucTFactor and DeepTFactor vary with different sequence redundancy, we compare them on datasets with sequence non-redundant (pairwise sequence similarity of < 30%) proteins, i.e., D 1 , D 2 , or D 3 , as well as on datasets with sequence redundant (pairwise sequence similarity of ≥ 30%) proteins, i.e., D 4 , D 5 , or D 6 . In general, because high sequence similarities in a dataset can artificially boost the performances of trained machine learning models (Hou et al, 2022), we expect to see an increase in the performance for both StrucTFactor and DeepTFactor with an increase in sequence redundancy, which is exactly what we find (Table 1). For example, with respect to MCC, the performance for DeepTFactor is ∼60% for D 1 (a sequence non-redundant dataset), while it is ∼96.5% for the corresponding sequence redundant dataset D 4 .s…”

Pitfalls of machine learning models for protein–protein interaction networks

“…In the absence of such studies, in silico PPI analyses are difficult to reconcile, the development of new models is inefficient, follow-up mechanisms studies are likely undermined and, ultimately, there are different versions of the underlying molecular networks that describe protein function. A range of publications have investigated best practices for machine learning in biology ( Chicco 2017 , Greener et al 2022 , Hou et al 2022 , Lee et al 2022 ) and highlighted that replicable, trustworthy, and generalizable high-performing models can capture more causal biology and enhance many aspects of biological research such as experimental designs and drug development.…”

Seq2Phase: language model-based accurate prediction of client proteins in liquid–liquid phase separation

“…Deep learning-based methods are effective for classification problems where such knowledge is not available in advance. Transformer, a deep-learning-based method that embeds natural language into vectors, was recently adopted for various classification problems in biology and has shown promising performances ( Vaswani et al 2017 , Hou et al 2022 ). Such language model-based tools include ProtTrans and ESM-2, which represent amino-acid sequences by vectors that can be used as inputs for various machine-learning methods ( Elnaggar et al 2022 , Lin et al 2023 ).…”