Curiosity in exploring chemical spaces: intrinsic rewards for molecular reinforcement learning

Abstract

Computer aided design of molecules has the potential to disrupt the field of drug and material discovery. Machine learning and deep learning in particular, made big strides in recent years and promises to greatly benefit computer aided methods. Reinforcement learning is a particularly promising approach since it enables de novo molecule design, that is molecular design, without providing any prior knowledge. However, the search space is vast, and therefore any reinforcement learning agent needs to perform efficient exploration. In this study, we examine three versions of intrinsic motivation to aid efficient exploration. The algorithms are adapted from intrinsic motivation in the literature that were developed in other settings, predominantly video games. We show that the \textit{curious} agents finds better performing molecules on two of three benchmarks. This indicates an exciting new research direction for reinforcement learning agents that can explore the chemical space out of their own motivation. This has the potential to eventually lead to unexpected new molecular designs no human has thought about so far.

ChemistGA: A Chemical Synthesizable Accessible Molecular Generation Algorithm for Real-World Drug Discovery

“…In drug discovery, experimental verification of molecular properties is very time-consuming and labor-intensive. Most DL generative models have not yet incorporated this challenge into the design process. − ,,− ,,− ,− To make ChemistGA more aware of this constraint by design, we propose an augmented ChemistGA, called R-ChemistGA, which simulates the real environment as further elucidated below.…”

On scientific understanding with artificial intelligence

“…It then chooses actions that lead to situations it cannot predict well, thus maximizing its own understanding of the environment. It has been shown using curious agents in simulated virtual universes 124 and robot agents in real laboratories 84 that curiosity is an efficient exploration strategy. Alternative intrinsic rewards for artificial agents are ‘computational creativity’ 125 , 126 and ‘surprise’ 127 .…”

A review of reinforcement learning in chemistry

“…As previously mentioned, the challenging syntax of SMILES strings makes their construction difficult. For this reason, Thiede et al 91 worked with SELFIES strings, in which every combination of characters is valid and the substrings generated during the construction process can be directly interpreted. Optimisation is handled via PPO, while the reward function is defined by a combination of an extrinsic reward (based on the predicted properties of the molecule) and an intrinsic reward named curiosity to encourage increased exploration of the state space.…”