Perplexity-based molecule ranking and bias estimation of chemical language models

Abstract

Chemical language models (CLMs) can be employed to design molecules with desired properties. CLMs generate new chemical structures in the form of textual representations, such as the simplified molecular input line entry systems (SMILES) strings, in a rule-free manner. However, the quality of these de novo generated molecules is difficult to assess a priori. In this study, we apply the perplexity metric to determine the degree to which the molecules generated by a CLM match the desired design objectives. This model-intrinsic score allows identifying and ranking the most promising molecular designs based on the probabilities learned by the CLM. Using perplexity to compare “greedy” (beam search) with “explorative” (multinomial sampling) methods for SMILES generation, certain advantages of multinomial sampling become apparent. Additionally, perplexity scoring is performed to identify undesired model biases introduced during model training and allows the development of a new ranking system to remove those undesired biases.

Exposing the limitations of molecular machine learning with activity cliffs.

“…Transfer learning 77 -applying a models' previously learned knowledge to a new, related problem by further training -was applied to the LSTM and transformer models in agreement with previous studies 64,66,78,79 . In a preliminary analysis, we explored transfer learning approaches for graph neural networks using self-supervision (context prediction 80 , infomax 81 , edge prediction 82 , and masking 80 ).…”

“…LSTMs -a type of recurrent neural network -can learn from string sequences by keeping track of long-range dependences. As in a previous study 64 , LSTM models were pre-trained on SMILES obtained by merging all training sets with no repetitions (36,281 molecules), using next-character prediction, before applying transfer learning for bioactivity prediction. 3.…”

“…Non-canonical SMILES strings were generated using RDKit 89 . (a) LSTM models were pre-trained on SMILES obtained by merging all training sets with no repetitions (36,281 molecules), using next-character prediction as in a recent study 64 . The network was composed of four layers comprising 5,820,515 parameters (layer 1: batch normalization; layer 2: LSTM with 1024 units; layer 3: LSTM with 256 units; layer 4: batch normalization).…”

“…For all models, we optimized the learning rate (lr), lr = [5x10 -4 , 5x10 -5 , or 5x10 -6 ]. The following hyperparameters were optimized: (a) GCN, hidden atom features (h a ), number of convolutional layers (n c ), hidden multiset transformer nodes (h t ), hidden predictor features (h p ), , h a = [32,64,128,256,512], n c = [1,2,3,4,5] [32,64,128,256]. All models were trained for 300 epochs, using early-stopping with a patience of 10 epochs.…”

Exposing the limitations of molecular machine learning with activity cliffs.

“…LSTMs -a type of recurrent neural network -can learn from string sequences by keeping track of long-range dependences. As in a previous study 68 , LSTM models were pre-trained on SMILES obtained by merging all training sets with no repetitions (36,281 molecules), using next-character prediction, before applying transfer learning for bioactivity prediction. 3.…”