Diverse scientific benchmarks for implicit membrane energy functions

Abstract

Energy functions are fundamental to biomolecular modeling. Their success depends on robust physical formalisms, efficient optimization, and high-resolution data for training and validation. Over the past 20 years, progress in each area has advanced soluble protein energy functions. Yet, energy functions for membrane proteins lag behind due to sparse and low-quality data, leading to overfit tools. To overcome this challenge, we assembled a suite of 12 tests on independent datasets varying in size, diversity, and resolution. The tests probe an energy function's ability to capture membrane protein orientation, stability, sequence, and structure. Here, we present the tests and use the franklin2019 energy function to demonstrate them. We then present a vision for transforming these "small" datasets into "big data" that can be used for more sophisticated energy function optimization. The tests are available through the Rosetta Benchmark Server (https://benchmark.graylab.jhu.edu/) and GitHub (https://github.com/rfalford12/Implicit-Membrane-Energy-Function-Benchmark).

Modeling membrane geometries implicitly in Rosetta

“…Therefore, we are able to test protein design including the pore for both energy functions, mpframework2012 and franklin2019. We used an established membrane protein design benchmark set to test where this improvement comes from [51].…”

“…The protein design analysis done is based on the MP Sequence Recovery benchmark test implemented within the Rosetta test server framework [51,67]. Since we were specifically looking at how inclusion of the pore affected the sequence recovery, we used a subset of the original dataset of only MPs that included a pore [68].…”

“…To dock glycophorin A we followed the steps described in Alford, Samanta, and Gray [51] for protein-protein docking using the franklin2019 energy function. The structure for glycophorin A was downloaded from both the Protein Data Bank (PDB) [70,71] and the OPM webserver [46].…”

Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks

“…We further want to directly and quickly compare different protocols and implementations and monitor the effect of score function changes onto the prediction results. For many years, Rosetta applications 30,31 and score functions [32][33][34][35] have been tested independently using the static benchmarking approach 19,36 , often with complete protocol captures 37,38 . The disadvantage of static benchmarking is that the results become outdated due to the lack of automation.…”