A deep generative model of the SARS-CoV-2 spike protein predicts future variants

Dhodapkar, Rahul

doi:10.1101/2023.01.17.524472

Cited by 4 publications

(7 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prot GPT2 uses the same tokenization algorithm as GPT2. Prot GPT2 has been applied in limited capacity to the Spike protein modeling problem [Dho23], for the purposes of generating individual mutations in the Receptor Binding Domain (RBD) sub-segment of the Spike protein. In this article, we present the first application of Prot GPT2 as a de novo generator of complete Spike protein sequences.…”

Section: Resultsmentioning

confidence: 99%

“…However, the problem of advance generation of the complete Spike protein sequence has not been studied before. While PLMs have been used to label Spike mutations with properties [HZBB21] [MBW + 22] or to generate individual mutations, or sub-sequences within certain regions of the Spike protein [Dho23] [CNW + 22], they have not been applied to the problem of advance generation of complete Spike protein sequences. Other machine learning methods have been developed to analyze the Spike protein, where they are used to characterize individual Spike mutations or effects of mutations on Spike sub-sequences [OJB + 22] [TWG + 22] [WLW + 23] [HLZ + 22].…”

Section: Introductionmentioning

confidence: 99%

“…In the case of protein sequence generation, quality is usually determined in silico [FSH22] [NRW + 22] or in vitro [SRK + 21] by measuring a small set of properties of the generated sequences in aggregate, such as sequence similarity, stability, and fold prediction. Prior generative work related to the Spike protein only evaluated individual mutations [Dho23], or generated sub-segments of the Spike protein carrying point mutations using methods that are not scalable to the full sequence generation task [CNW + 22] (where possible, we have included prior work in our experiments). Thus, the generated output is either a response to a prefix in the case of language (partial generation), not entirely correct or incorrect based on small discrete set membership (language, prior bioinformatics works), or only examines sub-sequences or individual mutations of the virus (partial generation).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

PandoGen: Generating complete instances of future SARS-CoV-2 sequences using Deep Learning

Ramachandran

Lumetta

Chen

2023

Preprint

View full text Add to dashboard Cite

Deep generative models have achieved state-of-the-art performance in many areas including image generation, code generation and natural language generation. We explore the use of deep generative models in producing complete instances of as-yet undiscovered SARS-CoV2 Spike protein sequences. The Spike protein is the primary initiator of infection by the SARS-CoV2 virus, and hence, the ability to predict future manifestations of the Spike protein is invaluable, enabling critical tasks such as advance validation of pharmaceutical interventions. We examine specific requirements of generating sequences for a pandemic and formulate a novel framework for training models for these requirements. Our solution only uses sequence information submitted in SARS-CoV2 repositories without the need for additional laboratory experiments. Resulting models substantially outperform a state-of-the-art generative model for protein sequences finetuned on SARS-CoV2 data. Samples produced from our models are four times as likely to be novel and real SARS-CoV2, and ten times as infectious, cumulatively. We find that among higher ranked sequences generated from our model, over 70% are discovered in the future, over twice the rate of the baseline. Our models represent a promising source of hypothetical SARS-CoV2 sequences, thus providing a key tool for advance preparation against the pandemic. PandoGen is available athttps://github.com/UIUC-ChenLab/PandoGen

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

PandoGen: Generating complete instances of future SARS-CoV-2 sequences using Deep Learning

Ramachandran

Lumetta

Chen

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In the case of protein sequence generation, quality is usually determined in silico [ 8 , 13 ] or in vitro [ 9 ] by measuring a small set of properties of the generated sequences in aggregate, such as sequence similarity, stability, and fold prediction. Prior generative work related to the Spike protein only evaluated individual mutations [ 16 ], or generated sub-segments of the Spike protein carrying point mutations using methods that are not scalable to the full sequence generation task [ 17 ] (where possible, we have included prior work in our experiments). Thus, the generated output is either a response to a prefix in the case of language (partial generation), not entirely correct or incorrect based on small discrete set membership (language, prior bioinformatics works), or only examines sub-sequences or individual mutations of the virus (partial generation).…”

Section: Introductionmentioning

confidence: 99%

PandoGen: Generating complete instances of future SARS-CoV-2 sequences using Deep Learning

Ramachandran,

Lumetta,

Chen

2024

PLoS Comput Biol

View full text Add to dashboard Cite

One of the challenges in a viral pandemic is the emergence of novel variants with different phenotypical characteristics. An ability to forecast future viral individuals at the sequence level enables advance preparation by characterizing the sequences and closing vulnerabilities in current preventative and therapeutic methods. In this article, we explore, in the context of a viral pandemic, the problem of generating complete instances of undiscovered viral protein sequences, which have a high likelihood of being discovered in the future using protein language models. Current approaches to training these models fit model parameters to a known sequence set, which does not suit pandemic forecasting as future sequences differ from known sequences in some respects. To address this, we develop a novel method, called PandoGen, to train protein language models towards the pandemic protein forecasting task. PandoGen combines techniques such as synthetic data generation, conditional sequence generation, and reward-based learning, enabling the model to forecast future sequences, with a high propensity to spread. Applying our method to modeling the SARS-CoV-2 Spike protein sequence, we find empirically that our model forecasts twice as many novel sequences with five times the case counts compared to a model that is 30× larger. Our method forecasts unseen lineages months in advance, whereas models 4× and 30× larger forecast almost no new lineages. When trained on data available up to a month before the onset of important Variants of Concern, our method consistently forecasts sequences belonging to those variants within tight sequence budgets.

show abstract

“…Training large language models from the ground up requires large datasets and vast computational resources, but pretrained models can be finetuned on specific protein families or properties. Dhodapkar (2023) introduced SpikeGPT2, a finetuned ProtGPT2 model that generates SARS-CoV-2 spike protein sequences to predict potential future mutations. Madani et al (2023) have also finetuned ProGen on protein families unseen during training.…”

Section: Introductionmentioning

confidence: 99%

Evaluating Tuning Strategies for Sequence Generation with Protein Language Models

Nathansen

Klein

Renard

et al. 2023

Preprint

View full text Add to dashboard Cite

Text generation models originally developed for natural language processing have proven to be successful in generating protein sequences. These models are often finetuned for improved performance on more specific tasks, such as generation of proteins from families unseen in training. Considering the high computational cost of finetuning separate models for each downstream task, prompt tuning has been proposed as an alternative. However, no openly available implementation of this approach compatible with protein language models has been previously published. Thus, we adapt an open-source codebase designed for NLP models to build a pipeline for prompt tuning on protein sequence data, supporting the protein language models ProtGPT2 and RITA. We evaluate our implementation by learning prompts for conditional sampling of sequences belonging to a specific protein family. This results in improved performance compared to the base model. However, in the presented use case, we observe discrepancies between text-based evaluation and predicted biological properties of the generated sequences, identifying open problems for principled assessment of protein sequence generation quality.

show abstract

A deep generative model of the SARS-CoV-2 spike protein predicts future variants

Cited by 4 publications

References 41 publications

PandoGen: Generating complete instances of future SARS-CoV-2 sequences using Deep Learning

PandoGen: Generating complete instances of future SARS-CoV-2 sequences using Deep Learning

PandoGen: Generating complete instances of future SARS-CoV-2 sequences using Deep Learning

Evaluating Tuning Strategies for Sequence Generation with Protein Language Models

Contact Info

Product

Resources

About