Double-head transformer neural network for molecular property prediction

Song, Yuanbing; Chen, Jinghua; Wang, Wenju; Chen, Gang; Ma, Zhichong

doi:10.1186/s13321-023-00700-4

Cited by 8 publications

(4 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many of the constructed ADMET Transformer models make use of those popularized in NLP literature, such as but not limited to BERT, RoBERTa, and GPT-2 [32][33][34][35][36][37][38][39][40][41][42]. Others combine graph representations with the Transformer to obtain graph-level contextual understanding [43][44][45][46][47][48][49]. In addition, some works use a combination of molecular line notation and pre-fabricated descriptors [38], while the remaining use the Transformer with various training strategies and architectural changes [50][51][52][53].…”

Section: Transformer-based Admet Modelsmentioning

confidence: 99%

Hybrid Fragment-SMILES Tokenization for ADMET Prediction in Drug Discovery

Aksamit,

Tchagang,

et al. 2024

Preprint

View full text Add to dashboard Cite

Background: Drug discovery and development is an extremely costly and time-consuming processing of identifying new molecules as therapeutics that can interact with a biomarker target to interrupt the disease pathway of interest. In addition to binding the target, a drug candidate needs to satisfy multiple properties affecting absorption, distribution, metabolism, excretion, and toxicity (ADMET). Computational and artificial intelligence approaches provide an opportunity to improve each step of the drug discovery and development process, in which the first question faced by us is how a molecule can be informatively represented such that the in-silico solutions are optimized. Results: This study introduces a novel hybrid SMILES-fragment tokenization method, coupled with two pre-training strategies, utilizing a Transformer-based model. We investigate the efficacy of hybrid tokenization in improving the performance of ADMET prediction tasks. Our approach leverages MTL-BERT, an encoder-only Transformer model that achieves state-of-the-art ADMET predictions, and contrasts the standard SMILES tokenization with our hybrid method across a spectrum of fragment library cutoffs. Conclusion: The findings reveal that while an excess of fragments can impede performance, using hybrid tokenization with high frequency fragments enhances results beyond the base SMILES tokenization. This advancement underscores the potential of integrating fragment- and character-level molecular features within the training of Transformer models for ADMET property prediction.

show abstract

Section: Transformer-based Admet Modelsmentioning

confidence: 99%

Hybrid Fragment-SMILES Tokenization for ADMET Prediction in Drug Discovery

Aksamit,

Tchagang,

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…This generative design approach successfully reproduced highly potent compounds for different activity classes based on weakly potent input compounds 13 . Transformer models have also been derived for other compound property predictions [14][15][16] and generative compound design applications [17][18][19] as well as for the prediction of drug-target interactions [20][21][22] .…”

Section: Meta-learning For Transformer-based Prediction Of Potent Com...mentioning

confidence: 99%

Meta-learning for transformer-based prediction of potent compounds

Chen,

Bajorath

2023

Sci Rep

View full text Add to dashboard Cite

For many machine learning applications in drug discovery, only limited amounts of training data are available. This typically applies to compound design and activity prediction and often restricts machine learning, especially deep learning. For low-data applications, specialized learning strategies can be considered to limit required training data. Among these is meta-learning that attempts to enable learning in low-data regimes by combining outputs of different models and utilizing meta-data from these predictions. However, in drug discovery settings, meta-learning is still in its infancy. In this study, we have explored meta-learning for the prediction of potent compounds via generative design using transformer models. For different activity classes, meta-learning models were derived to predict highly potent compounds from weakly potent templates in the presence of varying amounts of fine-tuning data and compared to other transformers developed for this task. Meta-learning consistently led to statistically significant improvements in model performance, in particular, when fine-tuning data were limited. Moreover, meta-learning models generated target compounds with higher potency and larger potency differences between templates and targets than other transformers, indicating their potential for low-data compound design.

show abstract

“…TranGRU , on the other hand, enhances the understanding of both local and global molecular information, positioning itself as a versatile sequence encoder for molecular representation extraction. DHTNN , a novel algorithmic framework, introduces the innovative Beaf activation function and leverages a Transformer with Double-head attention for molecular feature extraction, resulting in a robust approach that ensures model convergence and rational weight assignments [ 149 ]. Two strategies, MolHGT and PharmHGT , both of them applied the Heterogeneous Graph Transformer mechanism in molecular property research.…”

Section: Applications Of Attention-based Models In Drug Discoverymentioning

confidence: 99%

Attention is all you need: utilizing attention in AI-enabled drug discovery

Zhang,

Liu,

Liu

et al. 2023

Briefings in Bioinformatics

119

View full text Add to dashboard Cite

Recently, attention mechanism and derived models have gained significant traction in drug development due to their outstanding performance and interpretability in handling complex data structures. This review offers an in-depth exploration of the principles underlying attention-based models and their advantages in drug discovery. We further elaborate on their applications in various aspects of drug development, from molecular screening and target binding to property prediction and molecule generation. Finally, we discuss the current challenges faced in the application of attention mechanisms and Artificial Intelligence technologies, including data quality, model interpretability and computational resource constraints, along with future directions for research. Given the accelerating pace of technological advancement, we believe that attention-based models will have an increasingly prominent role in future drug discovery. We anticipate that these models will usher in revolutionary breakthroughs in the pharmaceutical domain, significantly accelerating the pace of drug development.

show abstract

Double-head transformer neural network for molecular property prediction

Cited by 8 publications

References 59 publications

Hybrid Fragment-SMILES Tokenization for ADMET Prediction in Drug Discovery

Hybrid Fragment-SMILES Tokenization for ADMET Prediction in Drug Discovery

Meta-learning for transformer-based prediction of potent compounds

Attention is all you need: utilizing attention in AI-enabled drug discovery

Contact Info

Product

Resources

About