Molecular Descriptors Property Prediction Using Transformer-Based Approach

Abstract

In this study, we introduce semi-supervised machine learning models designed to predict molecular properties. Our model employs a two-stage approach, involving pre-training and fine-tuning. Particularly, our model leverages a substantial amount of labeled and unlabeled data consisting of SMILES strings, a text representation system for molecules. During the pre-training stage, our model capitalizes on the Masked Language Model, which is widely used in natural language processing, for learning molecular chemical space representations. During the fine-tuning stage, our model is trained on a smaller labeled dataset to tackle specific downstream tasks, such as classification or regression. Preliminary results indicate that our model demonstrates comparable performance to state-of-the-art models on the chosen downstream tasks from MoleculeNet. Additionally, to reduce the computational overhead, we propose a new approach taking advantage of 3D compound structures for calculating the attention score used in the end-to-end transformer model to predict anti-malaria drug candidates. The results show that using the proposed attention score, our end-to-end model is able to have comparable performance with pre-trained models.

Predicting blood–brain barrier permeability of molecules with a large language model and machine learning

“…SMILES is a widely used standard for representing molecular structures as strings of characters that can be easily input into a transformer-based model 65 – 69 . Furthermore, using SMILES allows for greater flexibility and generalization of the input data because it can capture various molecular structures and properties 66 , 67 , 70 . This makes the transformer-based models more robust and adaptable to new and diverse sets of molecules, which are critical for new drug discovery 71 – 73 .…”