Image2SMILES: Transformer-based Molecular Optical Recognition Engine

Abstract

The rise of deep learning in various scientific and technology areas promotes the development of AI-based tools for information retrieval. Optical recognition of organic structures is a key part of the automated extraction of chemical information. However, this is a challenging task because there is a large variety of representation styles. In this research, we present a Transformer-based artificial neural network to convert images of organic structures to molecular structures. To train the model, we created a comprehensive data generator that stochastically simulates various drawing styles, functional groups, functional group placeholders (R-groups), and visual contamination. We demonstrate that the Transformer-based architecture can gather chemical insights from our generator with almost absolute confidence. That means that, with Transformer, one can fully concentrate on data simulation to build a good recognition model. A web demo of our optical recognition engine is available online at <i>Syntelly</i> platform.

SwinOCSR: end-to-end optical chemical structure recognition using a Swin Transformer

“…DECIMER 1.0 achieved a Tanimoto level of about 96% in a dataset of 30-35 million molecules. The latest deep-learning-based OCSR approach, Image2SMILES [16], uses ResNet-50 [27] as a backbone to extract image features and Transformer as a decoder part to predict FG-SMILES in the dataset of 10 million molecules. Image2SMILES achieved an accuracy of about 90.7%, but it still needs further improvement.…”

“…Compared with image captioning, two major challenges of an OCSR task are complex chemical patterns in chemical structures and long corresponding chemical representation. The existing methods based on deep learning [13][14][15][16] use CNNs as their backbones to extract image features of molecules. However, CNN only learns local representation and cannot effectively use global information.…”

DECIMER.ai - An open platform for automated optical chemical structure identification, segmentation and recognition in scientific publications

“…In recent years, deep-learning-based OCSR tools have been developed 16,17,18 in conjunction with remarkable advancements in computer vision and natural language processing 19,20 . While several publications have claimed to have developed tools that are capable of recognizing chemical depictions with high accuracy, most of these tools are either proprietary or entirely unavailable 16,[21][22][23] . Among the few open-source OCSR software solutions 15,24 , there is no system that combines chemical structure image segmentation, classification, and translation within a comprehensive workflow.…”

Performance of chemical structure string representations for chemical image recognition using transformers

“…In contrast, predicted SMILES may be invalid due to syntax errors such as mismatched binding symbols, branching, or ring closure. Other recent OCSR approaches 11–14 that used SMILES strings for output representation did not specifically address their inherent problems.…”