TransPolymer: a Transformer-based language model for polymer property predictions

Abstract

Accurate and efficient prediction of polymer properties is of great significance in polymer design. Conventionally, expensive and time-consuming experiments or simulations are required to evaluate polymer functions. Recently, Transformer models, equipped with self-attention mechanisms, have exhibited superior performance in natural language processing. However, such methods have not been investigated in polymer sciences. Herein, we report TransPolymer, a Transformer-based language model for polymer property prediction. Our proposed polymer tokenizer with chemical awareness enables learning representations from polymer sequences. Rigorous experiments on ten polymer property prediction benchmarks demonstrate the superior performance of TransPolymer. Moreover, we show that TransPolymer benefits from pretraining on large unlabeled dataset via Masked Language Modeling. Experimental results further manifest the important role of self-attention in modeling polymer sequences. We highlight this model as a promising computational tool for promoting rational polymer design and understanding structure-property relationships from a data science view.

Transferring a Molecular Foundation Model for Polymer Property Predictions

“…The fine-tuned models are evaluated based on their accuracy when predicting properties for polymers unseen in training. The evaluation metrics are relative root-mean-square error (RMSE) and R 2 with a 5-fold cross-validation, which are similar to those used for Trans-Polymer 15 and polyBERT. 16 The relative RMSE is defined as the RMSE scaled by the difference between the maximum and minimum values of each property.…”

“…In similar fashion to Kuenneth et al, 16,34 we employ a multitasking approach which predicts all properties simultaneously for better data efficiency. Our results show that the models pretrained on Enamine REAL achieve similar accuracy to the TransPolymer and polyBERT work, thereby suggesting that a model pretrained on a large-scale, accessible, and general molecular 4, 15 which has neither Eat nor STD values, and polyBERT is reproduced from Kuenneth and Ramprasad's Figure 5 and Table S1. 16 SML-MT, pretrained on small molecules, achieves similar accuracy to TransPolymer and polyBERT, pretrained on augmented polymer data sets.…”

Navigating the Expansive Landscapes of Soft Materials: A User Guide for High-Throughput Workflows

“…These graphs capture the composition and architecture of polydisperse polymers, and have improved prediction of polymer properties compared to molecular descriptors alone. , For example, they have been successful in predicting catalysis conditions for ring-opening polymerizations across multiple different data sets . TransPolymer, developed from the Transformer-based language processing algorithm, has been successfully trained on diverse polymer data sets for different bulk properties . Additionally, graph representation has been successful in theoretical studies, such as predicting the radius of gyration ( R g ) of coarse-grained model polymers with defined sequence and composition …”

“…Other types of dimensionality reduction tools include t-distributed stochastic neighbor embedding (t-SNE), designed to visualize high-dimensional data using a mapping onto two- or three-dimensions, and uniform manifold approximation and projection (UMAP), a topology driven technique . For example, the TransPolymer model uses t-SNE to visualize millions of unlabeled training data points and data from specific property databases . Many other types of dimension reduction techniques are detailed thoroughly by Banerjee and co-workers …”

“…84 TransPolymer, developed from the Transformer-based language processing algorithm, has been successfully trained on diverse polymer data sets for different bulk properties. 163 Additionally, graph representation has been successful in theoretical studies, such as predicting the radius of gyration (R g ) of coarse-grained model polymers with defined sequence and composition. 83 Sequence-defined monodisperse materials can leverage techniques used for biopolymers, which benefit from relatively limited composition, well-defined sequence, and training data sets such as the Protein Data Bank.…”