From archive to analysis: accessing web archives at scale through a cloud-based interface

Ruest, Nick; Fritz, Samantha; Deschamps, Ryan; Lin, Jimmy; Milligan, Ian

doi:10.1007/s42803-020-00029-6

Cited by 9 publications

(5 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The reasons for these differences need further investigation and research. As stated by Ruest et al (2021), who conducted research specifically on the web archiving tool named Heritrix, advanced knowledge of how its user needs the processing chain works. In addition, comprehension of the method to debug odd web behaviour during crawling process encounters by Heritrix is also important.…”

Section: Discussionmentioning

confidence: 99%

Exploration of Web Contents of Selangor Royal Family using Web Archiving Tools

Mohd Nor,

Saiful Bahry,

Mohamed Shuhidan

et al. 2024

E-BPJ

View full text Add to dashboard Cite

This study aims to explore the web contents of Selangor Royal family as an initiative to preserve the contents of social and cultural values. Initially, an exploration of websites’ profiles and the contents of Selangor Royal family by using tools BuildWith and XML-Sitemaps. Then, the selected web pages were then archived using web archiving tools, HTTrack and Conifer. The findings indicate that there are different types of websites and website owners covering the Selangor Royal family-related content. The recorded information and lifestyles of royal family represent value in building Malaysian identity and are significant to be preserved for future discovery.

show abstract

Section: Discussionmentioning

confidence: 99%

Exploration of Web Contents of Selangor Royal Family using Web Archiving Tools

Mohd Nor,

Saiful Bahry,

Mohamed Shuhidan

et al. 2024

E-BPJ

View full text Add to dashboard Cite

show abstract

“…While Archive-It collections can be accessed openly on the web using an interface similar to the Internet Archive Wayback Machine, AU provides a data analysis toolset compatible with Archive-It collections. Specifically, the Archives Unleashed Cloud service was in development at the time of my study, tailored for Archive-It users and organizations (Ruest et al, 2021). In November 2018, I acted as a participant-observer engaging with three teams during the two-day “datathon” event hosted in Vancouver.…”

Section: Methodsmentioning

confidence: 99%

All WARC and no playback: The materialities of data-centered web archives research

Maemura

2023

Big Data & Society

View full text Add to dashboard Cite

This paper examines the Web ARChive (WARC) file format, revealing how the format has come to play a central role in the development and standardization of interoperable tools and methods for the international web archiving community. In the context of emerging big data approaches, I consider the sociotechnical relationships between material construction of data and information infrastructures for collecting and research. Analysis is inspired by Star and Griesemer's historical case of the Museum of Vertebrate Zoology which reveals how boundary objects and methods standardization are used to enroll actors in the work of collecting for natural history. I extend these concepts by pairing them with frameworks for studying digital materiality and the representational qualities of data artifacts. Through examples drawn from fieldwork observations studying two data-centered research projects, I consider how the materiality of the WARC format influences research methods and approaches to data extraction, selection, and transformation. Findings identify three modalities researchers use to configure WARC data for researcher needs: using indexes to support search queries, constructing derivative formats designed for certain types of analysis, and generating custom-designed datasets tailored for specific research purposes. Findings additionally reveal similarities in how these distinct methods approach automated data extraction by relying upon the WARC's standardized metadata elements. By interrogating whose information needs are being met and taken into account in the design of the WARC's underlying information representation, I reveal effects on the emerging field of web history, and consider alternative approaches to knowledge production with archived web data.

show abstract

“…To this end, the project aspires to make petabytes of historical internet content accessible to scholars and others interested in researching the recent past. Between 2017 and 2020, the project focused on developing the "Archives Unleashed Cloud, " a web-based interface for working with web archives at scale using the Archives Unleashed Toolkit and Apache Spark [12]. This work built on the project's long-standing interests in building exploratory search interfaces for web archive collections [8].…”

Section: Related Work and Project Contextmentioning

confidence: 99%

“…2 The Archives Unleashed project aims to address this problem [13] by being for web archive analysis as Archive-It is for web archive capture: powerful, scalable, and above all, accessible and intuitive for users. The Archives Unleashed Cloud (2017-2020) provided user access to the features of the Archives Unleashed Toolkit in a cloud-hosted environment [12]. The Cloud worked with Archive-It collections, using APIs to transfer data from the Internet Archive to Compute Canada cloud-hosted infrastructure.…”

Section: Related Work and Project Contextmentioning

confidence: 99%

“…In this paper, we present the Archives Research Compute Hub (ARCH), a production system tightly integrated with the Internet Archive infrastructure and services. ARCH grew out of the Archives Unleashed Cloud: a proof-of-concept platform that demonstrated the ability of a web browser-based system to power backend Apache Spark-driven jobs on web archival datasets [12]. Powered by the Archives Unleashed Toolkit and the Internet Archive's Sparkling data processing library, the ARCH platform will become a complementary component of the Internet Archive's Archive-It system.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Abcdef

Holzmann

Ruest

Bailey

et al. 2022

Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries

Self Cite

View full text Add to dashboard Cite

Over the past quarter-century, web archive collection has emerged as a user-friendly process thanks to cloud-hosted solutions such as the Internet Archive's Archive-It subscription service. Despite advancements in collecting web archive content, no equivalent has been found by way of a user-friendly cloud-hosted analysis system. Web archive processing and research require significant hardware resources and cumbersome tools that interdisciplinary researchers find difficult to work with. In this paper, we identify six principlesthe ABCDEFs (Archive, Big data, Concurrent, Distributed, Efficient, and Flexible) -used to guide the development and design of a system. These make the transformation of, and working with, web archive data as enjoyable as the collection process. We make these objectives -largely common sense -explicit and transparent in this paper. They can be employed by every computing platform in the area of digital libraries and archives and adapted by teams seeking to implement similar infrastructures. Furthermore, we present ARCH (Archives Research Compute Hub) 1 , the first cloud-based system designed from scratch to meet all of these six key principles. ARCH is an interactive interface, closely connected with Archive-It, engineered to provide analytical actions, specifically generating datasets and in-browser visualizations. It efficiently streamlines research workflows while eliminating the burden of computing requirements. Building off past work by both the Internet Archive (Archive-It Research Services) and the Archives Unleashed Project (the Archives Unleashed Cloud), this merged platform achieves a scalable processing pipeline for web archive research. It is opensource and can be considered a reference implementation of the ABCDEF, which we have evaluated and discussed in terms of feasibility and compliance as a benchmark for similar platforms.

show abstract

From archive to analysis: accessing web archives at scale through a cloud-based interface

Cited by 9 publications

References 18 publications

Exploration of Web Contents of Selangor Royal Family using Web Archiving Tools

Exploration of Web Contents of Selangor Royal Family using Web Archiving Tools

All WARC and no playback: The materialities of data-centered web archives research

Abcdef

Contact Info

Product

Resources

About