A study on experimental bias in post-translational modification predictors

Abstract

Motivation: With a regulatory impact on numerous biological processes, protein phosphorylation is one of the most studied post-translational modifications. Effective computational methods that provide a sequence-based prediction of probable phosphorylation sites are desirable to guide functional experiments or constrain search spaces for proteomics-based experimental pipelines. Currently, the most successful methods train deep learning models on amino acid composition representations. However, recently proposed protein language models provide enriched sequence representations that may contain higher-level pattern information on which more performant phosphorylation site predictions may be based. Results: We explored the applicability of protein language models to general phosphorylation site prediction. We found that training prediction models on top of protein language models yield a relative improvement of between 13.4% and 63.3% in terms of area under the precision-recall curve over the state-of-the-art predictors. Advanced model interpretation and model transferability experiments reveal that across models, protease-specific cleavage patterns give rise to a protease-specific training bias that results in an overly optimistic estimation of phosphorylation site prediction performance, an important caveat in the application of advanced machine learning approaches to protein modification prediction based on proteomics data.

PhosBoost: Improved phosphorylation prediction recall using gradient boosting and protein language models

“…Therefore, the prediction of protein phosphosites is instrumental in elucidating possible regulatory phosphosites where extensive experimental phosphoproteomic data is not available (Meng et al, 2022). While the use of state-of-the-art deep learning and pLMs for protein phosphorylation prediction improved performance over existing methods (Zuallaert et al, 2022), classical machine learning approaches, such as gradient boosting trees, remain as powerful and scalable classification methods (Anghel et al, 2019;Lyashevska et al, 2021). Furthermore, ensemble methods that combine different machine-learning classification methods can improve classification performance by learning from multiple weak classifiers (Dietterich, 1997).…”

“…The PhosBoost achieved a higher recall score at all probability thresholds (Figure 1h). To provide additional support for using the PhosBoost stacking classifier approach, we conducted a similar analysis on an independent dataset, the Ramasamy22 protein phosphorylation dataset, obtained from the PhosphoLingo preprint (Zuallaert et al, 2022).…”

“…Conservation of phosphosites within and across species has been observed offering a complementary approach for improving phosphosite detection and annotation based on straightforward sequence similarity approaches (Amanchy et al, 2007 ; Chaudhuri et al, 2015 ; Maathuis, 2008 ; Tan et al, 2009 ). The large increase in the amount of experimental protein phosphorylation data facilitated the development of a variety of machine‐ and deep‐learning‐based protein phosphorylation classifications (Luo et al, 2019 ; Wang et al, 2017 ; Zuallaert et al, 2022 ). For example, classical machine learning algorithms such as random forests (Ismail et al, 2016 ; Liu et al, 2022 ; Wei et al, 2017 ), support vector machines (Dou et al, 2014 ; Jamal et al, 2021 ), and gradient boosting trees (Maiti et al, 2020 ) have been used for protein phosphorylation prediction.…”

“…For example, classical machine learning algorithms such as random forests (Ismail et al, 2016 ; Liu et al, 2022 ; Wei et al, 2017 ), support vector machines (Dou et al, 2014 ; Jamal et al, 2021 ), and gradient boosting trees (Maiti et al, 2020 ) have been used for protein phosphorylation prediction. More recently, deep learning algorithms such as convolutional neural networks (Guo et al, 2020 ; Luo et al, 2019 ; Wang et al, 2017 ; Zuallaert et al, 2022 ) and long short‐term memory networks (Lv et al, 2021 ; Thapa et al, 2021 ) enabled learning directly from protein sequences, making substantial improvements in protein phosphorylation prediction.…”

“…More recently, deep learning algorithms such as convolutional neural networks (Guo et al, 2020;Luo et al, 2019;Wang et al, 2017;Zuallaert et al, 2022) and long short-term memory networks (Lv et al, 2021;Thapa et al, 2021) enabled learning directly from protein sequences, making substantial improvements in protein phosphorylation prediction.…”