EXSCLAIM!: Harnessing materials science literature for self-labeled microscopy datasets

Materials science in the era of large language models: a perspective

“…† based approach, namely that nearly a third of the extracted images did not receive "a substring of text from the caption distributor", and that this "is where transformer-based NLP models stand to make the greatest contribution to the overall pipeline". 132 However, this increased exibility comes at the cost of potential hallucinations, lack of explainability, poor reproducibility and a larger computational overhead (explored further in Section 5). A synthesis of the two approaches, for example using classical NLP to ground LLM labels, could prove fruitful.…”

“…Image-based paper data extractors also exist, like ‘ImageDataExtractor’ 131 which is capable of detecting electron microscopy images from figures, identifying their scale and segmenting any nanoparticles present. ‘EXSCLAIM’ 132 uses rule-based NLP and image processing to extract images and assign hierarchical labels based on the figure caption. These extractors have found use in generating structured datasets for nanoparticles, 133 photocatalysts 134 and self-cleaning coatings.…”

“…This LLM-based workflow is potentially more flexible than approaches using classical NLP in that rules and desired data can be changed more easily and it is more robust to varying caption syntax – the authors of EXSCLAIM noted shortcomings in the rules-based approach, namely that nearly a third of the extracted images did not receive “a substring of text from the caption distributor”, and that this “is where transformer-based NLP models stand to make the greatest contribution to the overall pipeline”. 132…”