Predicting the Genotoxicity of Secondary and Aromatic Amines Using Data Subsetting To Generate a Model Ensemble

Abstract

Binary quantitative structure-activity relationship (QSAR) models are developed to classify a data set of 334 aromatic and secondary amine compounds as genotoxic or nongenotoxic based on information calculated solely from chemical structure. Genotoxic endpoints for each compound were determined using the SOS Chromotest in both the presence and absence of an S9 rat liver homogenate. Compounds were considered genotoxic if assay results indicated a positive genotoxicity hit for either the S9 inactivated or S9 activated assay. Each compound in the data set was encoded through the calculation of numerical descriptors that describe various aspects of chemical structure (e.g. topological, geometric, electronic, polar surface area). Furthermore, five additional descriptors that focused on the secondary and aromatic nitrogen atoms in each molecule were calculated specifically for this study. Descriptor subsets were examined using a genetic algorithm search engine interfaced with a k-Nearest Neighbor fitness evaluator to find the most information-rich subsets, which ultimately served as the final predictive models. Models were chosen for their ability to minimize the total number of misclassifications, with special attention given to those models that possessed fewer occurrences of positive toxicity hits being misclassified as nontoxic (false negatives). In addition, a subsetting procedure was used to form an ensemble of models using different combinations of compounds in the training and prediction sets. This was done to ensure that consistent results could be obtained regardless of training set composition. The procedure also allowed for each compound to be externally validated three times by different training set data with the resultant predictions being used in a "majority rules" voting scheme to produce a consensus prediction for each member of the data set. The individual models produced an average training set classification rate of 71.6% and an average prediction set classification rate of 67.7%. However, the model ensemble was able to correctly classify the genotoxicity of 72.2% of all prediction set compounds.

Novel approach to evolutionary neural network based descriptor selection and QSAR model development

“…Besides, LOO used as wrapper FF provides only internal model validation. Therefore internal and external validation of QSAR model based on single or multiple hold out test set prediction results became a standard tool for ENN/GNN training and performance evaluation in recent years [11,12,[25][26][27]. We do agree with reviewers comment that multiple leave-many-out (LMO) external validation could lead to underfitting due to removal of significant proportion of molecules from already small dataset [28].…”

“…It has been shown on a number of different QSAR problems that these wrappers could efficiently lower external test set and/or leave-one-out (LOO) prediction error [3][4][5][6][7][8][9][10][11][12]. Subject that has not been thoroughly documented before is final ENN/GNN QSAR model interpretation.…”

“…different random subsamplings [13] will also cause different final QSAR models i.e. description instability [11,14]. This behavior, also known as 'Rashomon effect' [15] is well-documented property of both, feature selection methods and neural networks (NN) [16], as well as other prediction tools like random forests [17].…”

“…This behavior, also known as 'Rashomon effect' [15] is well-documented property of both, feature selection methods and neural networks (NN) [16], as well as other prediction tools like random forests [17]. Many authors used model ensembles for improvement of external validation and proved adequacy of this approach [11,[16][17][18][19]. But such improvement of external validation results leads to very complex model interpretation [15,16].…”

“…But only multiple external validation results could provide realistic picture [13,16,23,24]. Random subsampling based on multiple partitions of the dataset into training and test set represents extensive application of cross-validation that has been introduced to QSAR by Mattioni et al [11] and to related fields by Leardi and Gonzales [20]. Stability of solutions against random subsampling could be referred to consistency of QSAR generalization results.…”