Chemformer: A Pre-Trained Transformer for Computational Chemistry

Abstract

Transformer models coupled with Simplified Molecular Line Entry System (SMILES) have recently proven to be a powerful combination for solving challenges in cheminformatics. These models, however, are often developed specifically for a single application and can be very resource-intensive to train. In this work we present Chemformer model -a Transformerbased model which can be quickly applied to both sequence-to-sequence and discriminative cheminformatics tasks. Additionally, we show that self-supervised pre-training can improve performance and significantly speed up convergence on downstream tasks. On direct synthesis and retrosynthesis prediction benchmark datasets we publish state-of-the-art results for top-1 accuracy. We also improve on existing approaches for a molecular optimisation task and show that Chemformer can optimise on multiple discriminative tasks simultaneously. Models, datasets and code will be made available after publication.

Fragment‐based drug discovery—the importance of high‐quality molecule libraries

“…Transformers extend this concept by using an ‘attention‐only’ framework that eliminates the need for RNN in sequence‐based tasks [ 62 ]. Recently, this newer approach has also been investigated for molecular optimisation and reaction prediction [ 63 ].…”

The rise of the machines in chemistry

“…Physical chemical properties, such as hydrophobicity (logP), solvent accessible area (SAS), chemical shifts, and scalar couplings, are common predictive tasks for small molecule focused deep learning applications. [34][35][36][37][38] The popularity of these tasks is aided by their many uses in drug discovery, particularly in the lead optimization stage where molecules must be optimized to simultaneously satisfy a number of criteria, such as solubility and bioavailability. Deep learning has also been applied to the prediction of NMR chemical shifts and scalar coupling constants, [39][40][41] collisional cross sections (CSS) in mass spectrometry, [42] and protein properties such as isoelectric point and fluorescence extinction coefficient.…”

“…The use of deep learning for the "generation" of molecules is one of the more high profile examples, with applications in lead optimization, protein structure prediction, and protein design. For small molecules, popular applications include the prediction of retrosynthetic pathways [38,[48][49][50] and the generation of novel molecules, either from a starting molecule, such as a lead compound, or de novo. [34,51,52] Finally, machine learned potentials have demonstrated that deep learning models can be successfully applied to atomistic simulations.…”

“…In addition to translation tasks, the vectors obtained from the encoder can be used as numerical feature inputs for training models which predict tasks such as molecular properties. [35,38] These models can be composed of a variety of different layer types, including graph convolutional layers and multi-F I G U R E 1 An example of a transformer model. This example contains both an encoder (left) and a decoder (right), although examples exist that only contain one of the two.…”

“…[94,95] This method has been successfully applied to transformer models for predictive tasks in cheminformatics, proteomics, and drug-target affinity prediction. [38,44,47]…”