Meta-learning for transformer-based prediction of potent compounds

Abstract

For many machine learning applications in drug discovery, only limited amounts of training data are available. This typically applies to compound design and activity prediction and often restricts machine learning, especially deep learning. For low-data applications, specialized learning strategies can be considered to limit required training data. Among these is meta-learning that attempts to enable learning in low-data regimes by combining outputs of different models and utilizing meta-data from these predictions. However, in drug discovery settings, meta-learning is still in its infancy. In this study, we have explored meta-learning for the prediction of potent compounds via generative design using transformer models. For different activity classes, meta-learning models were derived to predict highly potent compounds from weakly potent templates in the presence of varying amounts of fine-tuning data and compared to other transformers developed for this task. Meta-learning consistently led to statistically significant improvements in model performance, in particular, when fine-tuning data were limited. Moreover, meta-learning models generated target compounds with higher potency and larger potency differences between templates and targets than other transformers, indicating their potential for low-data compound design.

Chemical language models for molecular design

“…Different from chemical structure conversion via generative modeling, the inclusion of molecular property constraints in compound design requires the implementation of conditional transformers models. For example, such transformers were derived to predict activity cliffs [32] or individual highly potent compounds [33].…”

“…Following this approach, the DeepAC model predicted activity cliffs with an accuracy at least comparable to top‐performing machine learning models and further extended such predictions through generative design of new activity cliff compounds [32]. Furthermore, compound predictions conditioned on large potency differences were generalized beyond activity cliffs by adjusting the training protocol to predict highly potent compounds from weakly potent input templates [33]. The transformer CLM was shown to successfully predict known highly potent compounds from different activity classes not encountered during training.…”

“…The transformer CLM was shown to successfully predict known highly potent compounds from different activity classes not encountered during training. Moreover, the model was capable of creating highly potent compounds that were structurally diverse [33], as shown in Figure 3.…”

“…For each compound, its pK i value is reported and for each pair, the potency difference (figure adapted from Ref. [33]).…”

“…In addition, a CLM variant was implemented to enable generative design of highly potent compounds in low‐data regimes based on meta‐learning [34]. Using varying amounts of fine‐tuning data, the CLM meta‐learner consistently generated compounds with higher potency than a corresponding transformer without the meta‐learning extension [34].…”