Extracting Figures and Captions from Scientific Publications

Abstract

Figures and captions convey essential information in scientific publications. As such, there is a growing interest in mining published figures and in utilizing their respective captions as a source of knowledge. There is also much interest in image captioning systems that can automatically generate captions for images, whose training requires large datasets of image-caption pairs. Notably, the first fundamental step of obtaining figures and captions from publications is neither well-studied nor yet well-addressed. In this paper, we introduce a new and effective system for figure and caption extraction, PDFigCapX. Unlike current methods that extract figures by handling raw encoded contents of PDF documents, we separate text from graphical contents and utilize layout information to detect and disambiguate figures and captions. Files containing the figures and their associated captions are then produced as output to the enduser. We test PDFigCapX on both a previously used generic dataset and on two new sets of publications within the biomedical domain. Our experiments and results show a significant improvement in performance compared to the state-of-the-art, and demonstrate the effectiveness of our approach. Our system will be available for use at: https://www.eecis.udel.edu/~compbio/PDFigCapX.

Towards Understanding How Readers Integrate Charts and Captions: A Case Study with Line Charts

“…The prevalence of text with visuals has led researchers to explore how readers specifically understand information in figures with accompanying text in several domains. Li et al [25] conducted studies to demonstrate that figures with text can convey essential information and better aid understanding than just text alone for scientific publications in a biomedical domain. Odell et al [34] demonstrated that having text that accurately describes important findings in medical diagnostic images can increase physicians' speed and accuracy on Bayesian reasoning tasks while making life-critical judgments for patients.…”

ARCA. Semantic exploration of a bookstore

“…In future developments, the authors envision the implementation of new components to further enable multi-faceted content exploration. Such as the development of a storytelling component that connects geographic places and metadata to maps [2] and software that extracts images and caption from books [5] and integrate them into the ARCA knowledge graph.…”

MatGD: Materials Graph Digitizer

“…This is a substantial oversight, given that figures often encapsulate the critical results of a study, and these data are not often found in the text sections in a given publication. There are only a few cases where data extraction from figures in scientific papers has been attempted. − In particular, Nandy et al digitized thermalgravimetric analysis (TGA) data for approximately three thousand metal–organic framework papers. While their success is noteworthy, to expand this to a broader scope in materials science, there exists a pressing need to develop tools that can facilitate obtaining data found in figures.…”