Molecular Identification from AFM Images Using the IUPAC Nomenclature and Attribute Multimodal Recurrent Neural Networks

Abstract

Spectroscopic methods�like nuclear magnetic resonance, mass spectrometry, X-ray diffraction, and UV/visible spectroscopies�applied to molecular ensembles have so far been the workhorse for molecular identification. Here, we propose a radically different chemical characterization approach, based on the ability of noncontact atomic force microscopy with metal tips functionalized with a CO molecule at the tip apex (referred as HR-AFM) to resolve the internal structure of individual molecules. Our work demonstrates that a stack of constant-height HR-AFM images carries enough chemical information for a complete identification (structure and composition) of quasiplanar organic molecules, and that this information can be retrieved using machine learning techniques that are able to disentangle the contribution of chemical composition, bond topology, and internal torsion of the molecule to the HR-AFM contrast. In particular, we exploit multimodal recurrent neural networks (M-RNN) that combine convolutional neural networks for image analysis and recurrent neural networks to deal with language processing, to formulate the molecular identification as an imaging captioning problem. The algorithm is trained using a data set�which contains almost 700,000 molecules and 165 million theoretical AFM images�to produce as final output the IUPAC name of the imaged molecule. Our extensive test with theoretical images and a few experimental ones shows the potential of deep learning algorithms in the automatic identification of molecular compounds by AFM. This achievement supports the development of on-surface synthesis and overcomes some limitations of spectroscopic methods in traditional solution-based synthesis.

Mechanistic insights into on-surface reactions from isothermal temperature-programmed X-ray photoelectron spectroscopy

“…Given the widespread use of DFT calculated energy profiles for the rationalization of on-surface reactions, 44,51–53 benchmarking theoretical energy profiles with highly accurate experimental values is an extremely important task for validating prevalent theoretical approaches. The availability of a trustworthy, sufficiently large database – either from experiment or theory – may eventually facilitate the use of machine learning in surface chemistry, just as it is currently conquering the identification of molecules from their SPM images 54 and other areas of chemistry. 55…”

Autonomous Molecular Structure Imaging with High-Resolution Atomic Force Microscopy for Molecular Mixture Discovery

“…Second, there is work on using ML models to identify chemical structure and determine nomenclature of molecules imaged by HR-AFM. , Auto-HR-AFM’s script can be modified to integrate these molecular identification techniques. By collecting the image inputs needed by these techniques– multiple molecules at a series of different heights,– we can identify molecules by name in real time.…”

Atomic force microscopy simulations for CO-functionalized tips with deep learning

“…The combination of AFM imaging with Bayesian Inference and DFT calculations has been used to determine the adsorption configurations for a known molecule [46]. In previous work, we have demonstrated that it is possible to achieve a complete chemical identification of the structure and composition of a molecule from a 3D stack of constant-height HR-AFM images using two different approaches: (i) a Multimodal Recurrent Neural Network (M-RNN), that produces as output the IUPAC name of the molecule [47], and, (ii) a CGAN, that provides a 2D ball-and-stick model of the imaged molecule [48]. To the best of our knowledge, deep learning has not been used before to perform theoretical AFM simulations.…”

“…We find this analysis of a forward DL model for simulation of HR-AFM images to be also useful for better understanding how much physically relevant information about AFM imaging processes is encoded within the internal layers of the Convolutional Neural Network. This complementary view can help to further enhance inverse DL models, which are designed to extract the molecular structure from measured AFM data [42,47,48]. In particular, this forward model makes it easier to identify the most relevant data for the training, and to clearly spot information which is lost in the transformation.…”