Comparing the Influence of Simulated Experimental Errors on 12 Machine Learning Algorithms in Bioactivity Modeling Using 12 Diverse Data Sets

Cortés-Ciriano, Isidro; Bender, Andreas; Malliavin, Thérèse E.

doi:10.1021/acs.jcim.5b00101

Cited by 31 publications

(34 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Modelling the high confidence dataset led to a similar performance, with an RMSE test of 0.45 pGI 50 units and an R 2 0 test value of 0.84. This indicates that the predictive power of RF does not decrease when we included the data points measured in only one experiment, which is in agreement with a recent benchmarking study on the noise sensitivity of machine learning algorithms in bioactivity modelling ( Cortes-Ciriano et al. , 2015a ).…”

Section: Resultssupporting

confidence: 89%

Improved large-scale prediction of growth inhibition patterns using the NCI60 cancer cell line panel

et al. 2015

Self Cite

View full text Add to dashboard Cite

Motivation: Recent large-scale omics initiatives have catalogued the somatic alterations of cancer cell line panels along with their pharmacological response to hundreds of compounds. In this study, we have explored these data to advance computational approaches that enable more effective and targeted use of current and future anticancer therapeutics.Results: We modelled the 50% growth inhibition bioassay end-point (GI50) of 17 142 compounds screened against 59 cancer cell lines from the NCI60 panel (941 831 data-points, matrix 93.08% complete) by integrating the chemical and biological (cell line) information. We determine that the protein, gene transcript and miRNA abundance provide the highest predictive signal when modelling the GI50 endpoint, which significantly outperformed the DNA copy-number variation or exome sequencing data (Tukey’s Honestly Significant Difference, P <0.05). We demonstrate that, within the limits of the data, our approach exhibits the ability to both interpolate and extrapolate compound bioactivities to new cell lines and tissues and, although to a lesser extent, to dissimilar compounds. Moreover, our approach outperforms previous models generated on the GDSC dataset. Finally, we determine that in the cases investigated in more detail, the predicted drug-pathway associations and growth inhibition patterns are mostly consistent with the experimental data, which also suggests the possibility of identifying genomic markers of drug sensitivity for novel compounds on novel cell lines.Contact: terez@pasteur.fr; ab454@ac.cam.ukSupplementary information: Supplementary data are available at Bioinformatics online.

show abstract

Section: Resultssupporting

confidence: 89%

Improved large-scale prediction of growth inhibition patterns using the NCI60 cancer cell line panel

et al. 2015

Self Cite

View full text Add to dashboard Cite

show abstract

“…7). It is also important to consider that RF models are generally robust to moderate noise levels when modeling QSAR data sets, and hence, low levels of noise are well tolerated, and, in fact, might even help to generate models robust to noisy input data [62,63]. Together, these results indicate that although the predictions generated by moderately predictive base models might be noisy, they better explain the relevant variance connecting chemical and biological space [13], and that including base models with low predictive power does not add additional predictive signal to improve the modelling of these data sets.…”

Section: Resultsmentioning

confidence: 99%

QSAR-derived affinity fingerprints (part 2): modeling performance for potency prediction

et al. 2020

Self Cite

View full text Add to dashboard Cite

Affinity fingerprints report the activity of small molecules across a set of assays, and thus permit to gather information about the bioactivities of structurally dissimilar compounds, where models based on chemical structure alone are often limited, and model complex biological endpoints, such as human toxicity and in vitro cancer cell line sensitivity. Here, we propose to model in vitro compound activity using computationally predicted bioactivity profiles as compound descriptors. To this aim, we apply and validate a framework for the calculation of QSAR-derived affinity fingerprints (QAFFP) using a set of 1360 QSAR models generated using K i , K d , IC 50 and EC 50 data from ChEMBL database. QAFFP thus represent a method to encode and relate compounds on the basis of their similarity in bioactivity space. To benchmark the predictive power of QAFFP we assembled IC 50 data from ChEMBL database for 18 diverse cancer cell lines widely used in preclinical drug discovery, and 25 diverse protein target data sets. This study complements part 1 where the performance of QAFFP in similarity searching, scaffold hopping, and bioactivity classification is evaluated. Despite being inherently noisy, we show that using QAFFP as descriptors leads to errors in prediction on the test set in the ~ 0.65-0.95 pIC 50 units range, which are comparable to the estimated uncertainty of bioactivity data in ChEMBL (0.76-1.00 pIC 50 units). We find that the predictive power of QAFFP is slightly worse than that of Morgan2 fingerprints and 1D and 2D physicochemical descriptors, with an effect size in the 0.02-0.08 pIC 50 units range. Including QSAR models with low predictive power in the generation of QAFFP does not lead to improved predictive power. Given that the QSAR models we used to compute the QAFFP were selected on the basis of data availability alone, we anticipate better modeling results for QAFFP generated using more diverse and biologically meaningful targets. Data sets and Python code are publicly available at https ://githu b.com/isidr oc/QAFFP _regre ssion .

show abstract

“…The comparability of multiple independent bioactivity measurements (extended here to the area of analyzing cytotoxicity data) and the influence of data quality on bioactivity modeling have received increasing attention over the last few years . Many of these studies have used data derived from the ChEMBL database, which comprises compound activities against proteins, cell lines and complex systems such as whole organisms .…”

Section: Introductionmentioning

confidence: 99%

How Consistent are Publicly Reported Cytotoxicity Data? Large‐Scale Statistical Analysis of the Concordance of Public Independent Cytotoxicity Measurements

Cortés-Ciriano

Bender

2015

ChemMedChem

Self Cite

View full text Add to dashboard Cite

While increased attention is being paid to the impact of data quality in cell-line sensitivity and toxicology modeling, to date, no systematic study has evaluated the comparability of independent cytotoxicity measurements on a large-scale. Here, we estimate the experimental uncertainty of public cytotoxicity data from ChEMBL version 19. We applied stringent filtering criteria to assemble a curated data set comprised of pIC50 data for compound-cell line systems measured in independent laboratories. The estimated experimental uncertainty calculated was a mean unsigned error (MUE) value of 0.61-0.76, a median unsigned error (MedUE) value of 0.51-0.58, and a standard deviation of 0.76-1.00 pIC50 units. The experimental uncertainty (σE) estimated from all pairs of cytotoxicity measurements with a ΔpIC50 value lower than 2.5 was found to be 0.59-0.77 pIC50 units, and thus 21-60% and 21-26% higher than that of pKi and pIC50 data for ligand-protein data (σE =0.47-0.48 pKi units and σE =0.57-0.61 pIC50 units, respectively). The estimated σE value from the pairs of pIC50 values measured with metabolic assays was 0.98, whereas the σE value was found to be 0.69 when using the 1388 pIC50 pairs measured using exactly the same experimental setup. The maximum achievable Pearson correlation coefficient (RPearsonmax.2) of in silico models trained on cytotoxicity data from different laboratories was estimated to be 0.51-0.85, which is considerably different from the value of 1 corresponding to perfect predictions, hinting at the maximum performance one can expect also from computational cytotoxicity predictions. The lowest concordance between pairs of measurements was found for the drugs paclitaxel, methotrexate, zidovudine, and docetaxel, and for the cell lines HepG2, NCI-H460, L1210, and CCRF-CEM, hinting at particular sensitivity of those systems to experimental setups. The highest concordance was estimated for the compound-cell line system HL-60-etoposide (σE =0.70), whereas the lowest for L1210-methotrexate (σE =1.68). We found that annotation errors are responsible for the high discordance observed for some pairs of measurements, pointing out the importance of data curation when automatically extracting cytotoxicity data from public databases. Likewise, these results highlight the importance of estimating compound cytotoxicity with assays providing complementary biological information (i.e., metabolic, clonogenic and assays based on cell membrane integrity), especially when the mechanism of action of test compounds is unknown. From this analysis, guidelines can be created on the reliability of cytotoxicity data from public databases, which could ultimately prove valuable for modeling purposes, and to guide reporting of data in the literature.

show abstract

Comparing the Influence of Simulated Experimental Errors on 12 Machine Learning Algorithms in Bioactivity Modeling Using 12 Diverse Data Sets

Cited by 31 publications

References 66 publications

Improved large-scale prediction of growth inhibition patterns using the NCI60 cancer cell line panel

Improved large-scale prediction of growth inhibition patterns using the NCI60 cancer cell line panel

QSAR-derived affinity fingerprints (part 2): modeling performance for potency prediction

How Consistent are Publicly Reported Cytotoxicity Data? Large‐Scale Statistical Analysis of the Concordance of Public Independent Cytotoxicity Measurements

Contact Info

Product

Resources

About