PDBench: Evaluating Computational Methods for Protein Sequence Design

Abstract

Proteins perform critical processes in all living systems: converting solar energy into chemical energy, replicating DNA, as the basis of highly performant materials, sensing and much more. While an incredible range of functionality has been sampled in nature, it accounts for a tiny fraction of the possible protein universe. If we could tap into this pool of unexplored protein structures, we could search for novel proteins with useful properties that we could apply to tackle the environmental and medical challenges facing humanity. This is the purpose of de novo protein design. Sequence design is an important aspect of de novo protein design, and many successful methods to do this have been developed. Recently, deep-learning methods that frame it as a classification problem have emerged as a powerful approach. Beyond their reported improvement in performance, their primary advantage over physics-based methods is that the computational burden is shifted from the user to the developers, thereby increasing accessibility to the design method. Despite this trend, the tools for assessment and comparison of such models remain quite generic. The goal of this paper is to both address the timely problem of evaluation and to shine a spotlight, within the Machine Learning community, on specific assessment criteria that will accelerate impact. We present a carefully curated benchmark set of proteins and propose a number of standard tests to assess the performance of deep learning based methods. Our robust benchmark provides biological insight into the behaviour of sequencedesign methods, which is essential for evaluating their performance and practical utility. We compare five existing models with two novel models for sequence prediction. Finally, we test the designs produced by these models with Al-phaFold2, a state-of-the-art structure-prediction algorithm, to determine whether they are likely to fold into the intended 3-Dimensional shapes. BackgroundProteins are the molecules that perform almost all of the biochemical work in all living things. They have a staggering array of functionality from incredibly performant materials, like silks and wools, to some of the most efficient catalysts, capable of accelerating complex chemical reactions (Alberts et al. 2002). Beyond their roles in nature, proteins

From sequence to function through structure: Deep learning for protein design

“…CAFA [143] and CAGI [144] , however they focus on scoring how in-silico tools predict known function from sequence, rather than their ability to infer proteins (sequences or structures) that perform a desired, sometimes non-naturally found function. Conversely, model developers score their tools by metrics like Natural Sequence Recovery (NSR), which validate a model’s ability to link structure to sequence, but often not a model’s ability to generate diverse sequences fitting a desired structure [56] . A recent push in benchmarks scoring models’ ability to engineer proteins [99] , [145] , [146] , [147] , [148] highlights three aspects that protein design tools should strive to solve: 1) emulate laboratory conditions, i.e., extrapolate from very little available data; 2) set multiple function generalization goals, i.e, measure different aspects of function, with the intent of finding an optimal solution rather than maximizing any one metric; 3) focus on the ability to address out of observed data distributions, i.e., design proteins that achieve functions not observed in nature.…”

“…The performance of these methods is usually evaluated by native sequence recovery (NSR), i.e., the percentage of wild-type amino acids recovered for an input sequence by the design method. While this metric imposes some limitations, given that the identity percentage does not necessarily correlate with expression or functional levels [56] , it is nevertheless a convenient measure to evaluate how well the method recapitulates wild-type sequences. Some of the first attempts came from SPIN [57] and SPIN2 [58,p.…”

From sequence to function through structure: deep learning for protein design

“…CAFA [128] and CAGI [129], however they focus on scoring how in-silico tools predict known function from sequence, rather than their ability to infer proteins (sequences or structures) that perform a desired, sometimes non-naturally found function. Conversely, model developers score their tools by metrics like Natural Sequence Recovery (NSR), which validate a model's ability to link structure to sequence, but often not a model's ability to generate diverse sequences fitting a desired structure [53]. A recent push in benchmarks scoring models' ability to engineer proteins [130]- [134] highlights three aspects that protein design tools should strive to solve: 1) emulate laboratory conditions, i.e., extrapolate from very little available data; 2) set multiple function generalization goals, i.e, measure different aspects of function, with the intent of finding an optimal solution rather than maximizing any one metric; 3) focus on the ability to address out of observed data distributions, i.e., design proteins that achieve functions not observed in nature.…”

“…While this metric imposes some limitations, given that the identity percentage does not necessarily correlate with expression or functional levels [53] and the lack of common benchmark sequences, it is nevertheless a convenient measure to evaluate how well the method recapitulates wild-type sequences. Some of the first attempts came from SPIN [54] and SPIN2 [55, p. 2], which leveraged three-layered fully-connected neural networks (FNN) to learn from structural features embedded as a 1-dimensional (1D) tensor representing backbone torsion angles, local fragment-derived profiles, and global energy-based features.…”