ArchiveSpark

Abstract

Web archives are a valuable resource for researchers of various disciplines. However, to use them as a scholarly source, researchers require a tool that provides efficient access to Web archive data for extraction and derivation of smaller datasets. Besides efficient access we identify five other objectives based on practical researcher needs such as ease of use, extensibility and reusability.Towards these objectives we propose ArchiveSpark, a framework for efficient, distributed Web archive processing that builds a research corpus by working on existing and standardized data formats commonly held by Web archiving institutions. Performance optimizations in ArchiveSpark, facilitated by the use of a widely available metadata index, result in significant speed-ups of data processing. Our benchmarks show that ArchiveSpark is faster than alternative approaches without depending on any additional data stores while improving usability by seamlessly integrating queries and derivations with external tools.

The Case For Alternative Web Archival Formats To Expedite The Data-To-Insight Cycle

“…The weakness of this approach, however, is its dependency on an inefficient storage format. Because ArchiveSpark [27] is the only published example we are aware of that leverages a derivative format in batch web archive analysis, we use it as a surrogate to evaluate the performance impact of WARC derivatives. The results show the performance is only comparable to more efficient formats for transactional workloads.…”

“…Querying an ArchivesUnleashedToolkit application therefore requires repeated loading of the WARC files in full. ArchiveSpark [27], on the other hand, leverages CDX to selectively load WARC. Without sophisticated I/O scheduling, however, a full disk scan can still outperform many selective disk reads bundled together.…”

Exploration of Web Contents of Selangor Royal Family using Web Archiving Tools

“…The most found studies on web archiving centred around technical aspects in which the research involved a few numbers of technology tools or software. These can be referred to the study by Yu et al (2009), Chu et al (2010), Kawano (2011), Anjum (2012), Holzmann et al (2016), and Bahry et al (2019 that involved technology application as the research approach in which many of them used experimentation method. Previous studies also experimented with the best practices and technology tools to be used in any step involved in archiving the web collections.…”

Teaching Natural Language Processing through Big Data Text Summarization with Problem-Based Learning

“…Several Python libraries, such as NLTK (Bird et al, 2009), TextBlob (Loria et al, 2014), spaCy (Honnibal & Montani, 2017), and Gensim (Řehůřek & Sojka, 2011), can be used in linguistic analysis and clustering. MLlib (Meng et al, 2016), for efficient machine learning, and the ArchiveSpark (Holzmann, Goel, & Anand, 2016) library and extension of Apache Spark (2019), each facilitate processing on the Hadoop cluster. Regarding cloud computing and deep learning, we held a guest lecture and instructed students on how to install both TensorFlow (Abadi et al, 2016) and PyTorch (Paszke et al, 2019) and on how to run their code on the two ARC platforms.…”

“…As mentioned above, each event has two types of files: a data file (i.e., *.warc.gz) and an index file (i.e., *.cdx). We provided a Scala script built on ArchiveSpark (Holzmann et al, 2016) so that event teams can read the two files and extract Web page payloads for further processing. By sharing the Web archive files and our code, we expect students who are interested in Web archiving and big data to learn more about the accessible WARC format, Apache Spark, and parallel processing.…”