Diverse Scientific Benchmarks for Implicit Membrane Energy Functions

Abstract

Energy functions are fundamental to biomolecular modeling. Their success depends on robust physical formalisms, efficient optimization, and high-resolution data for training and validation. Over the past 20 years, progress in each area has advanced soluble protein energy functions. Yet, energy functions for membrane proteins lag behind due to sparse and low-quality data, leading to overfit tools. To overcome this challenge, we assembled a suite of 12 tests on independent data sets varying in size, diversity, and resolution. The tests probe an energy function’s ability to capture membrane protein orientation, stability, sequence, and structure. Here, we present the tests and use the franklin2019 energy function to demonstrate them. We then identify areas for energy function improvement and discuss potential future integration with machine-learning-based optimization methods. The tests are available through the Rosetta Benchmark Server () and GitHub ().

Modeling membrane geometries implicitly in Rosetta

“…Therefore, we are able to test protein design including the pore for both energy functions, mpframework2012 and franklin2019 . We used an established membrane protein design benchmark set to test where this improvement comes from (Alford et al, 2021 ). The test uses three different metrics to describe performance: (Alford et al, 2020 ) sequence recovery is the fraction of native residues recovered after design over all designable positions.…”

“…(Alford et al, 2015 ) non‐random recovery of individual amino acids, where the recovery rate for each amino acid is calculated relative to the background probability of randomly guessing the native amino acid (1/20). (Alford et al, 2021 ) The Kullback–Leibler (KL) divergence measures how different distributions of designed amino acids are compared to native distributions. The design test done on the subset of proteins from the dataset that contain a pore to highlight the effect of accounting for an aqueous pore.…”

“…The protein design analysis is based on the MP Sequence Recovery benchmark test implemented within the Rosetta test server framework (Alford et al, 2021 ; Koehler Leman et al, 2021 ). Since we were specifically looking at how inclusion of the pore affected the sequence recovery, we used a subset of the original dataset of only MPs that included a pore (Koehler Leman et al, 2017 ).…”

Implicit model to capture electrostatic features of membrane environment

“…Thus, a key challenge for energy function and the focus for our first and second tests is to recapitulate this lowest energy orientation of membrane peptides. Following our prior work [ 43 , 44 , 58 ], we used a protocol that samples all possible orientations of the peptide relative to the implicit membrane within ±60Å of the bilayer center (d), tilt angles relative to the membrane normal ( θ ) between ±180°, rotation angles relative to the principal helical axis ( ϕ ) between 0 − 360°. The global energy minimum of all sampled positions is defined as the most stable predicted orientation.…”

“…To confront the challenge of over-fitting and a specialized model, our group recently developed a 12-test benchmark suite that probes (1) protein orientation in the bilayer, (2) stability, (3) sequence, and (4) docking structures of membrane protein. [ 43 ] These tests form a platform to evaluate the strengths of an energy function and suggest areas of improvement. We evaluate the performance of F23 on predicting the (1) orientation of peptides in the membrane environment, (2) thermal stability due to point mutation, (3) transfer energy of peptides and (4) design evaluations.…”