The Giles Ecosystem – Storage, Text Extraction, and OCR of Documents

Abstract

In the digital humanities, there is a constant need to turn images and PDF files into plain text to apply analyses such as topic modelling, named entity recognition, and other techniques. However, although there exist different solutions to extract text embedded in PDF files or run OCR on images, they typically require additional training (for example, scholars have to learn how to use the command line) or are difficult to automate without programming skills. The Giles Ecosystem is a distributed system based on Apache Kafka that allows users to upload documents for text and image extraction. The system components are implemented using Java and the Spring Framework and are available under an Open Source license on GitHub (https://github.com/diging/).

Quantifying simultaneous innovations in evolutionary medicine

“…(Trevathan et al 1999;Gluckman et al 2009) For each individual we gather their academic publication from Web of Science (Reuters 2012) and transform these into plain text files and then into a comprehensive corpus using the Giles framework. (Damerow et al 2017) Our full corpus contains 6, 456 full-text publications from 1971 through 2017. The corpus is then hand curated to identify errors, such as wrongly assigning work to individuals.…”

Creating a Software Methodology to Analyze and Preserve Archaeological Legacy Data

“…Applying CTA to archaeological texts is much less common, but it has recently been used to analyze archaeological publications (e.g., Park et al 2020; Schmidt and Marwick 2020). Finally, OCR is often used as a digitizing tool by archaeologists (e.g., Heath et al 2019; McManamon et al 2017) and is also common in the digital humanities, along with text extraction (Damerow et al 2017; Pintus et al 2015). These spatial and textual analysis methods have been combined in archaeological research through what Murrieta-Flores and Gregory (2015) refer to as Geographic Text Analysis (GTA).…”