Evaluating Virtual Screening Methods:  Good and Bad Metrics for the “Early Recognition” Problem

Abstract

Many metrics are currently used to evaluate the performance of ranking methods in virtual screening (VS), for instance, the area under the receiver operating characteristic curve (ROC), the area under the accumulation curve (AUAC), the average rank of actives, the enrichment factor (EF), and the robust initial enhancement (RIE) proposed by Sheridan et al. In this work, we show that the ROC, the AUAC, and the average rank metrics have the same inappropriate behaviors that make them poor metrics for comparing VS methods whose purpose is to rank actives early in an ordered list (the "early recognition problem"). In doing so, we derive mathematical formulas that relate those metrics together. Moreover, we show that the EF metric is not sensitive to ranking performance before and after the cutoff. Instead, we formally generalize the ROC metric to the early recognition problem which leads us to propose a novel metric called the Boltzmann-enhanced discrimination of receiver operating characteristic that turns out to contain the discrimination power of the RIE metric but incorporates the statistical significance from ROC and its well-behaved boundaries. Finally, two major sources of errors, namely, the statistical error and the "saturation effects", are examined. This leads to practical recommendations for the number of actives, the number of inactives, and the "early recognition" importance parameter that one should use when comparing ranking methods. Although this work is applied specifically to VS, it is general and can be used to analyze any method that needs to segregate actives toward the front of a rank-ordered list.

<strong>Virtual screening tailored ensembles of QSAR models for the discovery of dual A_2A Adenosine Receptor Antagonists / Monoamine Oxidase B Inhibitors</strong>

“…The model with the highest value of BEDROC (α=160.9) among all single models within the ensemble is also the one with the best value of this metric among the set of representative valid models. This improvement might seem meaningless, however it can be interpreted as 5% http://sciforum.net/conference/mol2net-1 more probability of retrieving a dual target ligand in the first 1% of the ranked list using the obtained ensemble compared to the best performing individual model [22]. From Table 2 it can also be seen that the values of BEDROC for α=32.2 and α=20 are also higher when the ensemble model is compared with its members.…”

“…al. [22] In our previous research [9] the VS-tailored ensembles were obtained using only a very limited set of predictive models that were selected as the best ones derived from each modeling approach. However, during the modeling process a larger number of high quality models are obtained that we did not considered for ensemble modeling in that investigation.…”

Maximum Common Substructure-Based Data Fusion in Similarity Searching

“…We have therefore chosen to use the Boltzman-Enhanced ROC score (BEDROC) as an alternative evaluation score. BEDROC scales from 0.0 to 1.0, where a value closer to 1.0 indicates superior virtual screening performance 31 .…”

Impact of Benchmark Data Set Topology on the Validation of Virtual Screening Methods: Exploration and Quantification by Spatial Statistics

“…[15][16][17][18] However, the ROC metric has one important shortcoming: it is unable to address the so-called "early recognition" problem. 13,14 Since usually only a small fraction of a VS ranking can be tested experimentally, a good metric for VS should reflect the enrichment of actives at the beginning of the ranking. Recent efforts have sought to develop VS performance metrics that combine the statistic stability of ROC with the early recognition properties of RTR or EF.…”

“…Recent efforts have sought to develop VS performance metrics that combine the statistic stability of ROC with the early recognition properties of RTR or EF. 13,14 However, due to their novelty, these metrics have not yet found extensive use in VS validation studies and it is therefore difficult to compare the results obtained with these metrics to those of other works. Thus, in this study, RTR and ROC were used in a complementary manner for the analysis of VS rankings.…”