Generating stable molecules using imitation and reinforcement learning

Abstract

Chemical space is routinely explored by machine learning methods to discover interesting molecules, before time-consuming experimental synthesizing is attempted. However, these methods often rely on a graph representation, ignoring 3D information necessary for determining the stability of the molecules. We propose a reinforcement learning approach for generating molecules in cartesian coordinates allowing for quantum chemical prediction of the stability. To improve sample-efficiency we learn basic chemical rules from imitation learning on the GDB-11 database to create an initial model applicable for all stoichiometries. We then deploy multiple copies of the model conditioned on a specific stoichiometry in a reinforcement learning setting. The models correctly identify low energy molecules in the database and produce novel isomers not found in the training set. Finally, we apply the model to larger molecules to show how reinforcement learning further refines the imitation learning model in domains far from the training data.

A review of reinforcement learning in chemistry

“…Recently, Meldgaard et al 103 proposed a method combining imitation learning and reinforcement learning to ensure that the generated molecules are stable. Imitation learning is a learning scheme in which an ML model may learn from demonstrations and has some history of use alongside RL 104 .…”

Assessing Deep Generative Models in Chemical Composition Space

“…Nonetheless, RL has also been applied for the design of molecules and materials. 28,30,[81][82][83][84] In this case, the agent sequentially adds atoms or functional groups to a system. The main challenge associated with this is that the reward (e.g.…”

Inverse design of 3d molecular structures with conditional generative neural networks

“…This includes specifically designed approaches to translate given molecular graphs to 3d conformations [32][33][34][35][36][37][38], map from coarse-grained to fine-grained structures [39], sample unbiased equilibrium configurations of a given system [40,41], or focus on protein folding [42][43][44][45][46]. In contrast, other models aim at sampling directly from distributions of 3d molecules with arbitrary composition [47][48][49][50][51][52][53][54][55][56], making them suitable for general inverse design settings. These models need to be biased towards structures with properties of interest, e.g.…”

“…These models need to be biased towards structures with properties of interest, e.g. using reinforcement learning [51,52,56], fine-tuning on a biased data set [48], or other heuristics [54]. Some of us have previously proposed G-SchNet [48], an auto-regressive deep neural network that generates diverse, small organic molecules by placing atom after atom in Euclidean space.…”