A Benchmark of PDF Information Extraction Tools Using a Multi-task and Multi-domain Evaluation Framework for Academic Documents

Abstract

Extracting information from academic PDF documents is crucial for numerous indexing, retrieval, and analysis use cases. Choosing the best tool to extract specific content elements is difficult because many, technically diverse tools are available, but recent performance benchmarks are rare. Moreover, such benchmarks typically cover only a few content elements like header metadata or bibliographic references and use smaller datasets from specific academic disciplines. We provide a large and diverse evaluation framework that supports more extraction tasks than most related datasets. Our framework builds upon DocBank, a multi-domain dataset of 1.5M annotated content elements extracted from 500K pages of research papers on arXiv. Using the new framework, we benchmark ten freely available tools in extracting document metadata, bibliographic references, tables, and other content elements from academic PDF documents. GROBID achieves the best metadata and reference extraction results, followed by CERMINE and Science Parse. For table extraction, Adobe Extract outperforms other tools, even though the performance is much lower than for other content elements. All tools struggle to extract lists, footers, and equations. We conclude that more research on improving and combining tools is necessary to achieve satisfactory extraction quality for most content elements. Evaluation datasets and frameworks like the one we present support this line of research. We make our data and code publicly available to contribute toward this goal.

The ACL OCL Corpus: Advancing Open Science in Computational Linguistics

“…After an extensive comparison with open-source toolkits such as PDFBox 5 and pdfminer.six 6 , we use GROBID 7 for the fulltext extraction from PDF files. Our study validates findings from Meuschke et al (2023) which found GROBID outperforms the other freely-available tools in metadata, reference, and general text extraction tasks from academic PDF documents. We take the S2ORC-JSON format used by and Wang et al (2020) for our full-text schema, which includes complete information parsed from PDF files, such as metadata, authors, and body text with citations, references, sections and etc.…”

LaTeX Rainbow: Universal LaTeX to PDF Document Semantic & Layout Annotation Framework

“…PyMuPDF 13 allows access to information about the more underlying details of the PDF file. However, a benchmark demonstrates their imperfect performance (Meuschke et al, 2023).…”

“…and multimodal contents such as images, tables, equations etc. and their captions (Meuschke et al, 2023;Bast and Korzen, 2017). Additionally, these documents often contain elements that are not directly related to the core content, such as watermarks (Chia et al, 2018), publisher details and header information that serves navigation in collections.…”

“…However, machine learning, being a data-driven approach, requires extensive labeled data. Considering that existing PDF extraction tools cannot guarantee the accuracy of the extraction (Meuschke et al, 2023). In this context, L A T E X code has emerged as a valuable resource.…”