Molecule generation using transformers and policy gradient reinforcement learning

Abstract

Generating novel valid molecules is often a difficult task, because the vast chemical space relies on the intuition of experienced chemists. In recent years, deep learning models have helped accelerate this process. These advanced models can also help identify suitable molecules for disease treatment. In this paper, we propose Taiga, a transformer-based architecture for the generation of molecules with desired properties. Using a two-stage approach, we first treat the problem as a language modeling task of predicting the next token, using SMILES strings. Then, we use reinforcement learning to optimize molecular properties such as QED. This approach allows our model to learn the underlying rules of chemistry and more easily optimize for molecules with desired properties. Our evaluation of Taiga, which was performed with multiple datasets and tasks, shows that Taiga is comparable to, or even outperforms, state-of-the-art baselines for molecule optimization, with improvements in the QED ranging from 2 to over 20 percent. The improvement was demonstrated both on datasets containing lead molecules and random molecules. We also show that with its two stages, Taiga is capable of generating molecules with higher biological property scores than the same model without reinforcement learning.

Chemical language models for molecular design

“…The design of virtual compound libraries via machine translation was one of the first applications of CLMs, for which RNNs [2,5,12,13] and, subsequently, transformers were used [13,14]. In addition, transformer networks have been employed for other generative compound design applications [15][16][17]. Furthermore, both RNN and transformer models have been extensively used for chemical reaction prediction [18][19][20][21], representing a growth area for CLM applications.…”

Offline RL for generative design of protein binders

“…Given the necessity to generate molecules with specific properties, the application of RL for drug design has gained traction in recent years [28,6,39,41,2,26]. These RL-based approaches have demonstrated efficacy in optimizing one or multiple chemical properties that are computationally inexpensive to evaluate.…”

“…Previous approaches to generating drugs have leveraged a variety of neural network architectures. These encompass variants of Variational Autoencoders (VAE) [13, 16], Recurrent Neural Networks (RNN) [28], Graph Neural Networks (GNN) [42], Generative Adversarial Networks (GAN) [14], and Transformers [3, 26].…”

“…Some of these tools are dedicated to assessing molecular properties [25,38], enabling the validation of molecules before embarking on costly real-world experiments. Another avenue of research focuses on generating molecules themselves [13,16,26]. Many such approaches rely on deep learning techniques, necessitating differentiability for training.…”

“…RL has exhibited promising results across a wide range of domains, including robotics [43] and Natural Language Processing (NLP) [29]. Online RL techniques have also found application in drug generation tasks [26]. However, the necessity for direct interaction with the environment can be a substantial bottleneck for RL algorithms, given the associated costs and potential risks, particularly when algorithms require thousands or even millions of interactions.…”