Figure and Figure Caption Extraction for Mixed Raster and Vector PDFs: Digitization of Astronomical Literature with OCR Features

Abstract

Scientific articles published prior to the "age of digitization" in the late 1990s contain figures which are "trapped" within their scanned pages. While progress to extract figures and their captions has been made, there is currently no robust method for this process. We present a YOLO-based method for use on scanned pages, post-Optical Character Recognition (OCR), which uses both grayscale and OCR-features. When applied to the astrophysics literature holdings of the Astrophysics Data System (ADS), we find F1 scores of 90.9% (92.2%) for figures (figure captions) with the intersection-over-union (IOU) cut-off of 0.9 which is a significant improvement over other state-of-the-art methods.

Generalizability in Document Layout Analysis for Scientific Article Figure & Caption Extraction

“…However, as shown by the solid and dashed orange lines of Figure 1, this is not the case. While the F1 score for the detectron2 model is higher than ours for figures, it does less well for figure captions (though, as discussed in [12] this may be due to inconsistencies between "caption" and "text" classes, the latter being the only class [12,14] (blue lines) and detectron2 [16] trained on the PubLayNet document dataset [17] (orange lines). Neither model generalizes well to the HathiTrust set of documents.…”

“…Our prior work was aimed at the extraction of figures and their captions from a subset of the "pre-digital" astrophysical literature holdings of the Astrophysics Data System (ADS) 1 using both grayscale and optical character recognition (OCR) features of article pages [12]. Our model produced a high level of accuracy on our dataset -for an intersection-over-union (IOU) metric of 0.9 we found F1 scores of ≥ 90% 2 [12,14].…”

“…A change in not only publication type, but simply publication year can drastically lower the accuracy of page object extraction methods for models that are not explicitly trained on this type of document [10,12,14]. Our prior work was aimed at the extraction of figures and their captions from a subset of the "pre-digital" astrophysical literature holdings of the Astrophysics Data System (ADS) 1 using both grayscale and optical character recognition (OCR) features of article pages [12]. Our model produced a high level of accuracy on our dataset -for an intersection-over-union (IOU) metric of 0.9 we found F1 scores of ≥ 90% 2 [12,14].…”

“…To give our model the best chance of success in our comparison, we subset this collection with the search fields of "astronomy" and filter the results to include only English-language Conference, Journal and Manuscript documents with "Full View" available, bringing the total records to 56,282. For our illustration, we select ≈350 randomly chosen pages from six articles within this filtered collection which we annotate with figure and caption class definitions from [12,14] in Make-Sense.ai 4 [15]. The precision, recall and F1 score at different IOU cut-offs for our model applied to this data are shown by solid and dashed navy blue lines in Figure 1.…”

“…In the cases of borndigital documents, these deep learning methods are combined with heuristically-derived results in post-processing steps [9]. However, for historical scanned documents, these methods present many challenges [10][11][12].…”