A Unified Framework of Scaffold-Lab for Critical Assessment of Protein Backbone Generation Methods

Abstract

De novo protein design has undergone a rapid development in recent years, especially for backbone generation, which stands out as more challenging yet valuable, offering the ability to design novel protein folds with fewer constraints. However, a comprehensive delineation of its potential for practical application in protein engineering remains lacking, as does a standardized evaluation framework to accurately assess the diverse methodologies within this field. Here, we proposed Scaffold-Lab benchmark focusing on evaluating unconditional generation across metrics like designability, novelty, diversity, efficiency and structural properties. We also extrapolated our benchmark to include the motif-scaffolding problem, demonstrating the utility of these conditional generation models. Our findings reveal that FrameFlow and RFdiffusion in unconditional generation and GPDL-H in conditional generation showcased the most outstanding performances. Furthermore, we described a systematic study to investigate conditional generation and applied it to the motif-scaffolding task, offering a novel perspective for the analysis and development of conditional protein design methods. All data and scripts are available at https://github.com/Immortals-33/Scaffold-Lab.