PTM-Mamba: A PTM-Aware Protein Language Model with Bidirectional Gated Mamba Blocks

Abstract

Proteins serve as the workhorses of living organisms, orchestrating a wide array of vital functions. Post-translational modifications (PTMs) of their amino acids greatly influence the structural and functional diversity of different protein types and uphold proteostasis, allowing cells to swiftly respond to environmental changes and intricately regulate complex biological processes. To this point, efforts to model the complex features of proteins have involved the training of large and expressive protein language models (pLMs) such as ESM-2 and ProtT5, which accurately encode structural, functional, and physicochemical properties of input protein sequences. However, the over 200 million sequences that these pLMs were trained on merely scratch the surface of proteomic diversity, as they neither input nor account for the effects of PTMs. In this work, we fill this major gap in protein sequence modeling by introducing PTM tokens into the pLM training regime. We then leverage recent advancements in structured state space models (SSMs), specifically Mamba, which utilizes efficient hardware-aware primitives to overcome the quadratic time complexities of Transformers. After adding a comprehensive set of PTM tokens to the model vocabulary, we train bidirectional Mamba blocks whose outputs are fused with state-of-the-art ESM-2 embeddings via a novel gating mechanism. We demonstrate that our resultant PTM-aware pLM,PTM-Mamba, improves upon ESM-2’s performance on various PTM-specific tasks. PTM-Mamba is the first and only pLM that can uniquely input and represent both wild-type and PTM sequences, motivating downstream modeling and design applications specific to post-translationally modified proteins. To facilitate PTM-aware protein language modeling applications, we have made our model available at:https://huggingface.co/ChatterjeeLab/PTM-Mamba.

Programmable Protein Stabilization with Language Model-Derived Peptide Guides

“…More interestingly, as a genetically-encoded tool, peptide-guided duAbs, alongside uAbs, may together comprise a powerful proteome screening tool for drug discovery, allowing for combinatorial protein activation and inhibition screening, similar to the CRISPR-based Perturb-Seq platform. 35 Finally, with recent advancements in protein representation algorithms, we envision that our language model-generated peptides can be augmented to specifically bind post-translational and mutant isoforms of target proteins, 36,37 and can be fused to other PTM domains, including kinases, phosphatases, and deglycosylases, to name a few. 7 This study, enabling modular peptide-guided protein stabilization, represents a next step towards this eventual goal of a fully programmable proteome editing system.…”

De NovoDesign of Peptide Binders to Conformationally Diverse Targets with Contrastive Language Modeling

“…We are also exploring alternative generative processes, such as techniques leveraging databases of antibody CDRs or human protein motifs and autoregressive models, such as GPT, Llama2, and Mamba, [47][48][49] and even extending to post-translationally modified (PTM) target sequences using PTM-aware pLM embeddings. 50 For future work, we plan to expand our in silico screening strategies to optimize other key properties, including selectivity and binding affinity to undruggable proteins via fine-tuning on specific target classes, such as fusion oncoproteins. 51 In total, our two-step method represents a first protein binder design module from target sequence alone, without the requirement of 3D structural information.…”

ProtMamba: a homology-aware but alignment-free protein state space model

“…Approaches such as HyenaDNA (Nguyen et al, 2023) or Evo (Nguyen et al, 2024) were trained on long DNA sequences and capture regulatory mechanics. Meanwhile, PTM-Mamba addresses post-translational modifications of protein sequences (Peng et al, 2024).…”