A novel applicability domain technique for mapping predictive reliability across the chemical space of a QSAR: reliability-density neighbourhood

Abstract

The ability to define the regions of chemical space where a predictive model can be safely used is a necessary condition to assure the reliability of new predictions. This implies that reliability must be determined across chemical space in the attempt to localize “safe” and “unsafe” regions for prediction. As a result we devised an applicability domain technique that addresses the data locally instead of handling it as a whole—the reliability-density neighbourhood (RDN). The main novelty aspect of this method is that it characterizes each single training instance according to the density of its neighbourhood in the training set, as well as its individual bias and precision. By scanning through the chemical space (by iteratively increasing the applicability domain area), it was observed that new test compounds are successively included into the applicability domain region in such a manner that strongly correlates to their predictive performance. This allows the mapping of local reliability across different locations in the training set space, and thus allows identifying regions where the model has low reliability. This method also showed matching profiles between two external sets, which is an indication that it performs robustly with new data. Another novel aspect in this technique is that it is paired with a specific feature selection algorithm. As a result, the impact of the feature set used was studied from which the top 20 features selected by ReliefF yielded the best results, as opposed to using the model’s features or the entire feature set as commonly done. As the third novel aspect, in this work we propose a new scoring function to help evaluate the quality of an applicability domain profile (i.e., the curve of accuracy vs the applicability domain measure in question). Overall, the RDN showed to be a promising method that can correctly sort new instances according to predictive performance. As a result, this technique can be received by an end-user as proof of concept for the performance of a QSAR model in new data, thus promoting the user’s trust on the QSAR output.Graphical abstract. Electronic supplementary materialThe online version of this article (doi:10.1186/s13321-016-0182-y) contains supplementary material, which is available to authorized users.

A visual approach for analysis and inference of molecular activity spaces

“…In QSAR modelling, KDE is usually explored as an interpolation method to define the applicability domain of generated classification models [45]. Among the most widely used multivariate (high dimensional metric space) interpolation approaches (e.g., range-based, distance-based, geometrical), KDE is considered as one of the more advanced and accurate methods for calculating the applicability domain [42–44]. However, to the best of our knowledge, KDE is not used for visualization nor data classification over 2D activity landscapes.…”

“…KDE is considered a powerful tool in statistics for truthful assessment of data actual distribution/characteristics [40]. In cheminformatics literature KDE has been used as a robust method to define applicability domains of quantitative structure-activity relationship (QSAR) predictive models [42–44]. Applicability domains are used to define a boundaries in molecular space within which new predictions of QSAR models are considered reliable [45].…”

In silico toxicity prediction, synthesis, characterization, antimicrobial and antioxidant activity of different di substituted chalcones

“…Antibacterial action of the combined mixes was tried against five bacterial strains and three parasitic strains utilizing agar well dispersion technique [15][16][17][18]. Dimethyl sulfoxide was utilized as dissolvable control.…”

Assessment and Reproducibility of Quantitative Structure–Activity Relationship Models by the Nonexpert

“…Methods for assessing applicability domain are discussed elsewhere. [45][46][47][48][49][50] This then moves some way towards recognising the difference between the average performance of a model for a test set (typically the modeller's measure of success) and the specific performance for a particular compound of interest (normally the user's measure of success). Successful navigation of the complete workflow illustrated in Figure 1 indicates a model has been shown to be useful, reproducible and predictive.…”