defoe: A Spark-Based Toolbox for Analysing Digital Historical Textual Data

Filgueira, Rosa; Ardanuy, Mariona Coll; Colavizza, Giovanni; Hetherington, James; Terras, Melissa; Jackson, Michael; Roubickova, Anna; Krause, Amrey; Ahnert, Ruth; Hauswedell, Tessa; Nyhan, Julianne; Beavan, David; Hobson, Timothy

doi:10.1109/escience.2019.00033

Cited by 7 publications

(9 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…See Figure 5. The new NLS object model is added to our previous collection of object models [4]. We have used it to analyse the Encyclopaedia Britannica, the Gazetteers of Scotland, and the Chapbooks printed in Scotland, as described in Section VI.…”

Section: A Ingestion Of Nls Digital Collectionsmentioning

confidence: 99%

See 1 more Smart Citation

Extending defoe for the Efficient Analysis of Historical Texts at Scale

Filgueira

Grover

Karaiskos

et al. 2021

2021 IEEE 17th International Conference on eScience (eScience)

Self Cite

View full text Add to dashboard Cite

This paper presents the new facilities provided in defoe, a parallel toolbox for querying a wealth of digitised newspapers and books at scale. defoe has been extended to work with further Natural Language Processing () tools such as the Edinburgh Geoparser, to store the preprocessed text in several storage facilities and to support different types of queries and analyses. We have also extended the collection of XML schemas supported by defoe, increasing the versatility of the tool for the analysis of digital historical textual data at scale. Finally, we have conducted several studies in which we worked with humanities and social science researchers who posed complex and interested questions to large-scale digital collections. Results shows that defoe allows researchers to conduct their studies and obtain results faster, while all the large-scale text mining complexity is automatically handled by defoe.

show abstract

Section: A Ingestion Of Nls Digital Collectionsmentioning

confidence: 99%

“…This section gives an overview of our previous work on defoe [4], providing the necessary background for the functionalities presented in Section IV.…”

Section: Introductionmentioning

confidence: 99%

Extending defoe for the Efficient Analysis of Historical Texts at Scale

Filgueira

Grover

Karaiskos

et al. 2021

2021 IEEE 17th International Conference on eScience (eScience)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The results indicated a significant improvement in the accuracy of Naive Bayes and Logistic Regression with respect to Decision Trees. The research presented in [23] introduced a new portable and scalable digital eScience toolbox called "defoe" that can extract knowledge from historical data by running text-mining queries across large historical datasets in parallel using Apache Spark. The experiments were done using five different historical datasets and two different high-performance computing (HPC) environments as well as on desktop [23].…”

Section: Research On Apache Sparkmentioning

confidence: 99%

“…The research presented in [23] introduced a new portable and scalable digital eScience toolbox called "defoe" that can extract knowledge from historical data by running text-mining queries across large historical datasets in parallel using Apache Spark. The experiments were done using five different historical datasets and two different high-performance computing (HPC) environments as well as on desktop [23]. The authors showed that "defoe" can allow researchers to query multiple datasets in parallel from a single command-line interface [23].…”

Section: Research On Apache Sparkmentioning

confidence: 99%

“…The experiments were done using five different historical datasets and two different high-performance computing (HPC) environments as well as on desktop [23]. The authors showed that "defoe" can allow researchers to query multiple datasets in parallel from a single command-line interface [23]. The authors in [24] implemented a collaborative filtering (CF) recommender system on Apache Spark that can draw on users' past opinions for the process of prediction generation.…”

Section: Research On Apache Sparkmentioning

confidence: 99%

See 1 more Smart Citation

A Parallel Processing Technique for Filtering and Storing User Specified Data

Chanda¹

View full text Add to dashboard Cite

Users are often interested in a specific type of data (user-preferred data) from a largevolume dataset. An efficient system that only stores user-preferred data from the large dataset can reduce the search latency, which allows the users to search for relevant information in a timely manner. The motivation behind this thesis is to devise a technique that filters a large dataset and stores only the filtered data, thereby saving storage space for the user. Running the filtering operation can be CPU-intensive, which can lead to high latency in extracting preferred data from the dataset. To solve this problem, the technique employs parallel processing and machine learning. A proof-of-concept prototype for this technique has been built on Apache Spark. The performance of the prototype subjected to synthetic datasets is analyzed. The analysis of experimental results shows the viability of this technique and provides insights into the system behavior and performance.

show abstract

Collaborative Historical Research in the Age of Big Data

Ahnert

Griffin

Ridge

et al. 2023

Self Cite

View full text Add to dashboard Cite

Living with Machines is the largest digital humanities project ever funded in the UK. The project brought together a team of twenty-three researchers to leverage more than twenty-years' worth of digitisation projects in order to deepen our understanding of the impact of mechanisation on nineteenth-century Britain. In contrast to many previous digital humanities projects which have sought to create resources, the project was concerned to work with what was already there, which whilst straightforward in theory is complex in practice. This Element describes the efforts to do so. It outlines the challenges of establishing and managing a truly multidisciplinary digital humanities project in the complex landscape of cultural data in the UK and share what other projects seeking to undertake digital history projects can learn from the experience. This title is also available as Open Access on Cambridge Core.

show abstract

defoe: A Spark-Based Toolbox for Analysing Digital Historical Textual Data

Cited by 7 publications

References 7 publications

Extending defoe for the Efficient Analysis of Historical Texts at Scale

Extending defoe for the Efficient Analysis of Historical Texts at Scale

A Parallel Processing Technique for Filtering and Storing User Specified Data

Collaborative Historical Research in the Age of Big Data

Contact Info

Product

Resources

About