Simpler is Better: How Linear Prediction Tasks Improve Transfer Learning in Chemical Autoencoders

Abstract

Transfer learning is a subfield of machine learning that leverages proficiency in one or more prediction tasks to improve proficiency in a related task. For chemical property prediction, transfer learning models represent a promising approach for addressing the data scarcity limitations of many properties by utilizing potentially abundant data from one or more adjacent applications. Transfer learning models typically utilize a latent variable that is common to several prediction tasks and provides a mechanism for information exchange between tasks. For chemical applications, it is still largely unknown how correlation between the prediction tasks affects performance, the limitations on the number of tasks that can be simultaneously trained in these models before incurring performance degradation, and if transfer learning positively or negatively affects ancillary model properties. Here we investigate these questions using an autoencoder latent space as a latent variable for transfer learning models for predicting properties from the QM9 data set that have been supplemented with semiempirical quantum chemistry calculations. We demonstrate that property prediction can be counterintuitively improved by utilizing a simpler linear predictor model, which has the effect of forcing the latent space to organize linearly with respect to each property. In data scarce prediction tasks, the transfer learning improvement is dramatic, whereas in data rich prediction tasks, there appears to be little adverse impact of transfer learning on prediction performance. The transfer learning approach demonstrated here thus represents a highly advantageous supplement to property prediction models with no downside in implementation.

Locally Optimizable Joint Embedding Framework to Design Nitrogen‐rich Molecules that are Similar but Improved

“…In contrast to earlier joint training approaches that were not designed for scarce data, [25,29] or are based on pretraining using chemical properties and a multitask transfer learning approach for target properties that are scarce, [27,28] the current multi‐class approach finds a joint embedding with manual feature vectors which therefore finds the latent space using structure information instead of properties. The augmentation is thus done using the low‐level structural information while the properties between the small and large data sets remain distinct and possibly non‐overlapping.…”

“…It has been shown capable of learning underlying relationships from a diverse set of molecular data by letting multiple data tasks and domains interact adaptively while generating the joint embeddings. Joint training is also a type of joint embedding that has been successfully applied to improve deep learning‐based molecule generation and enable transfer learning [25–29] . Joint training incorporates a property prediction task into a variational autoencoder (VAE) [30] and has been shown to organize points in the VAE latent space, making the latent space amenable to inverse molecular design and optimization [25,29] .…”

“…Iovanac and Savoie employed multitask joint training in which a data abundant property prediction task for (de)protonation energy, was combined with the target task predicting molecule acidity, [27] for which data are scarce. It therefore has been reasoned that joint embedding yields more expressive models [31] in part because of the likely organization they induce in the latent representation [25,28] . This, in turn, offers improved properties of molecules obtained from such generative models [29] …”

Actively Searching: Inverse Design of Novel Molecules with Simultaneously Optimized Properties

“…We utilize the grammar variational autoencoder 37 (GVAE) to achieve generation of molecules with targeted properties, with the alteration of using a single linear predictor layer so that properties tend to vary linearly along the latent dimensions. 38 Training was conducted using the RMSprop algorithm with a learning rate of 0.001, which was set to decay by a factor of 0.3 in the case of a plateau in the validation loss. KL prediction task based on a linear prediction network ensures that those properties will vary linearly along the principal components of the latent encodings.…”

“…KL prediction task based on a linear prediction network ensures that those properties will vary linearly along the principal components of the latent encodings. 38 Compounds with specific properties can then be generated by targeting regions of the latent space based on univariate linear regression between the property of interest and the position along one of the principle components. In the case of multi-property models, correlation between the properties may lead to latent space organization not being exactly orthogonal.…”