Construction of balanced, chemically dissimilar training, validation and test sets for machine learning on molecular datasets

Tricarico, Giovanni; Hofmans, Johan; Lenselink, Eelke B.; López-Ramos, Miriam; Dréanic, Marie-Pierre; Stouten, Pieter F. W.

doi:10.26434/chemrxiv-2022-m8l33-v2

Cited by 5 publications

(15 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, on average the median R 2 is 0.4 lower and the median RMSE is 0.2 higher for the DGBC split than the random split showing that the models do not perform as well on this data split. These results are in line with expectations and previously published results 21,38 and show the importance to assess model performance with a realistic split.…”

Section: Importance Of Data Splittingsupporting

confidence: 92%

“…Dissimilarity-driven balanced cluster (DGBC) split was made by using a method developed by Tricarico et al 21 First the compounds in the dataset were clustered using sphere exclusion clustering on ECFP6 fingerprints with a Tanimoto distance of 0.736 between cluster centroids. 32 Fingerprint generation and sphere exclusion clustering were done using RDKit (version 2020.09.05).…”

Section: Data Set Creationmentioning

confidence: 99%

“…For single-task modelling, this is often straightforward, but for multi-task modelling, the construction of realistic balanced train-validation-test splits without data leakage between tasks is not as straightforward. 21 If the splits are done per target, this will lead to data leakage, i.e. the same compound can be in the train for one task and in the validation or test set for another one.…”

Section: Introductionmentioning

confidence: 99%

“…Two out of the three challenges (balance and no data leakage) to create a good train-validation-test split for multi-task modelling can be addressed with a random global equilibrated selection (RGES) which is introduced in this paper. All three of them (no data-leakage, balance and dissimilarity) can be addressed with a dissimilarity-driven global balanced clustering (DGBC) split recently proposed by Tricarico et al 21 It simultaneously maximises dissimilarity and balances the individual sets globally.…”

Section: Introductionmentioning

confidence: 99%

“…28 We also propose two well-balanced 80-10-10 multi-task splits for activity prediction models: one based on random allocation and the other one on clustering. 21 Finally, we benchmarked the capacity of different approaches to utilise large-scale kinase activity data to build prediction models. This was done by comparing the performance of a set of single-task models and multi-task models with and without data imputation.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Large-scale modelling of sparse kinase activity data

Luukkonen

Meijer

Tricarico³

et al. 2023

Preprint

View full text Add to dashboard Cite

Protein kinases are a protein family that play an important role in several complex diseases such as cancer, cardiovascular and immunological diseases. Kinases have conserved binding sites, which when targeted can lead to similar activities of inhibitors against different kinases. This can be exploited to create multi-target drugs. On the other hand, selectivity (lack of similar activities) is desirable in order to avoid toxicity issues. There is a vast amount of kinase activity data in the public domain, which can be used in many different ways. Multi-task machine learning models are expected to excel for these kinds of datasets because they can learn from implicit correlations between tasks (in this case activities against a variety of kinases). However, multi-task modelling of sparse data poses two major challenges: (i) creating a balanced train-test split without data leakage and (ii) handling missing data. In this work, we construct a kinase benchmark set composed of two balanced splits without data leakage, using random and dissimilarity-driven cluster-based mechanisms, respectively. This data set can be used for benchmarking and developing kinase activity prediction models. Overall, the performance on the dissimilarity-driven cluster-based split is lower than on random splits based sets for all models, indicating poor generalizability of models. Nevertheless, we show that multi-task deep learning models, on this very sparse dataset, outperform single-task deep learning and tree-based models. Finally, we demonstrate that data imputation does not improve the performance of (multitask) models on this benchmark set.

show abstract

Section: Importance Of Data Splittingsupporting

confidence: 92%

Section: Data Set Creationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Large-scale modelling of sparse kinase activity data

Luukkonen

Meijer

Tricarico³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

Large-Scale Modeling of Sparse Protein Kinase Activity Data

Luukkonen

Meijer

Tricarico

et al. 2023

J. Chem. Inf. Model.

Self Cite

View full text Add to dashboard Cite

Protein kinases are a protein family that plays an important role in several complex diseases such as cancer and cardiovascular and immunological diseases. Protein kinases have conserved ATP binding sites, which when targeted can lead to similar activities of inhibitors against different kinases. This can be exploited to create multitarget drugs. On the other hand, selectivity (lack of similar activities) is desirable in order to avoid toxicity issues. There is a vast amount of protein kinase activity data in the public domain, which can be used in many different ways. Multitask machine learning models are expected to excel for these kinds of data sets because they can learn from implicit correlations between tasks (in this case activities against a variety of kinases). However, multitask modeling of sparse data poses two major challenges: (i) creating a balanced train–test split without data leakage and (ii) handling missing data. In this work, we construct a protein kinase benchmark set composed of two balanced splits without data leakage, using random and dissimilarity-driven cluster-based mechanisms, respectively. This data set can be used for benchmarking and developing protein kinase activity prediction models. Overall, the performance on the dissimilarity-driven cluster-based split is lower than on random split-based sets for all models, indicating poor generalizability of models. Nevertheless, we show that multitask deep learning models, on this very sparse data set, outperform single-task deep learning and tree-based models. Finally, we demonstrate that data imputation does not improve the performance of (multitask) models on this benchmark set.

show abstract

QSPRpred: a Flexible Open-Source Quantitative Structure-Property Relationship Modelling Tool

van den Maagdenberg,

Šícho,

Araripe

et al. 2024

J Cheminform

View full text Add to dashboard Cite

Building reliable and robust quantitative structure–property relationship (QSPR) models is a challenging task. First, the experimental data needs to be obtained, analyzed and curated. Second, the number of available methods is continuously growing and evaluating different algorithms and methodologies can be arduous. Finally, the last hurdle that researchers face is to ensure the reproducibility of their models and facilitate their transferability into practice. In this work, we introduce QSPRpred, a toolkit for analysis of bioactivity data sets and QSPR modelling, which attempts to address the aforementioned challenges. QSPRpred’s modular Python API enables users to intuitively describe different parts of a modelling workflow using a plethora of pre-implemented components, but also integrates customized implementations in a “plug-and-play” manner. QSPRpred data sets and models are directly serializable, which means they can be readily reproduced and put into operation after training as the models are saved with all required data pre-processing steps to make predictions on new compounds directly from SMILES strings. The general-purpose character of QSPRpred is also demonstrated by inclusion of support for multi-task and proteochemometric modelling. The package is extensively documented and comes with a large collection of tutorials to help new users. In this paper, we describe all of QSPRpred’s functionalities and also conduct a small benchmarking case study to illustrate how different components can be leveraged to compare a diverse set of models. QSPRpred is fully open-source and available at https://github.com/CDDLeiden/QSPRpred.Scientific ContributionQSPRpred aims to provide a complex, but comprehensive Python API to conduct all tasks encountered in QSPR modelling from data preparation and analysis to model creation and model deployment. In contrast to similar packages, QSPRpred offers a wider and more exhaustive range of capabilities and integrations with many popular packages that also go beyond QSPR modelling. A significant contribution of QSPRpred is also in its automated and highly standardized serialization scheme, which significantly improves reproducibility and transferability of models.

show abstract

Construction of balanced, chemically dissimilar training, validation and test sets for machine learning on molecular datasets

Cited by 5 publications

References 0 publications

Large-scale modelling of sparse kinase activity data

Large-scale modelling of sparse kinase activity data

Large-Scale Modeling of Sparse Protein Kinase Activity Data

QSPRpred: a Flexible Open-Source Quantitative Structure-Property Relationship Modelling Tool

Contact Info

Product

Resources

About