ImageDataExtractor: A Tool To Extract and Quantify Data from Microscopy Images

Abstract

The rise of data science is leading to new paradigms in data-driven materials discovery. This carries an essential notion that large data sources containing chemical structure and property information can be mined in a fashion that detects and exploits structure–property relationships, such that chemicals can be predicted to suit a given material application. The success of material predictions is predicated on these large data sources of chemical structure and property information being suited to a target application. Microscopy is commonly used to characterize chemical structure, especially in fields such as nanotechnology where material properties are highly dependent on the size and shape of nanoparticles. Large data sources of nanoparticle information stemming from microscopy images would thus be highly beneficial. Millions of microscopy images exist, but they lie fragmented across the literature, typically presented individually within a paper article and usually in a qualitative fashion therein, even though they harbor a wealth of numeric information. We present the ImageDataExtractor toolkit that autoidentifies and autoextracts microscopy images from scientific documents, whereupon it autonomously analyzes each image to produce quantitative particle size and shape information about its subject material. Each image is quantified by decoding its scale bar information using optical character recognition, with help from super-resolution convolutional neural networks where required. Individual particles are detected and profiled using various thresholding, segmentation, polygon fitting, and edge correction routines. The high-throughput operational capability of ImageDataExtractor means that it can be used to generate large-data sources of particle information for data-driven materials discovery. Evaluation metrics, precision and recall, are greater than 80% for the majority of the image processing steps, and precision is above 80% for all critical steps. The ImageDataExtractor tool is released under the MIT license and is available to download from .

Artificial intelligence and machine learning in glass science and technology: 21 challenges for the 21st century

“…Tools such as the caption‐cluster plot mentioned in Challenge 15 is very useful in extracting and identifying the labels of an image. We need packages such as the ImageDataExtractor 87 to extract and quantify images from the literature. However, the currently available tools are limited to optical microscopy or electron microscopy images and do not extend to other characterizations and spectra.…”

Process Discovery through Assimilation of Complex Biogeochemical Datasets

“…Similarly, the active data mining effort of RES 3 T (https://www.hzdr.de/db/RES3T.login) contains 3172 references and includes reactions between 147 minerals and 148 ligands and a total of 7062 reaction constants that span across all known surface complexation models. ML-empowered natural language processing and automated data discovery/extraction from diverse literatureincreasingly used in chemical and material sciences 4,5,[7][8][9][10][11] can similarly transform data utilization in the Earth sciences to improve ESP.…”

Looking through glass: Knowledge discovery from materials science literature using natural language processing

“…Very few studies have, however, focused on extracting information related to images and plots in literature. 23 , 24 The adage, “a picture is worth a thousand words,” is even more relevant to scientific literature, as images hold the most crucial information related to scientific hypothesis and theories. 25 Till date, there has been no framework that allows direct search or compilation of images presented in scientific literature.…”