Archiving the web using page changes patterns: a case study

Saâd, Myriam Ben; Gançarski, Stéphane

doi:10.1007/s00799-012-0094-z

Cited by 18 publications

(19 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Various efforts currently exist to capture the web at the local, national, international as well as institutional or community levels [50,62,78]. Studies of these efforts center around more efficient methods for capturing the dynamic web and increasing discoverability of the archived web by examining tools, policies, and metadata [19,38,67].…”

Section: Literature Reviewmentioning

confidence: 99%

Bots, Seeds and People

Summers

Punzalan

2017

Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing

View full text Add to dashboard Cite

The field of web archiving provides a unique mix of human and automated agents collaborating to achieve the preservation of the web. Centuries old theories of archival appraisal are being transplanted into the sociotechnical environment of the World Wide Web with varying degrees of success. The work of the archivist and bots in contact with the material of the web present a distinctive and understudied CSCW shaped problem. To investigate this space we conducted semi-structured interviews with archivists and technologists who were directly involved in the selection of content from the web for archives. These semi-structured interviews identified thematic areas that inform the appraisal process in web archives, some of which are encoded in heuristics and algorithms. Making the infrastructure of web archives legible to the archivist, the automated agents and the future researcher is presented as a challenge to the CSCW and archival community.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Bots, Seeds and People

Summers

Punzalan

2017

Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing

View full text Add to dashboard Cite

show abstract

“…Ben Saad and Gançarski performed a similar study regarding the importance of changes on a page [7][8][9]. Gray and Martin created a framework for high-quality mementos and assessing their quality by measuring the missing embedded resources [19].…”

Section: Related Workmentioning

confidence: 99%

“…8 When using a page-at-a-time archival service, the resulting memento contains embedded resources with the same archival datetime [1]. This section identifies our damage measurement of this page-at-a-time archiver and outlines the differences between Heritrix and WebCite.…”

Section: Measuring Webcitementioning

confidence: 99%

Not all mementos are created equal: measuring the impact of missing resources

et al. 2015

View full text Add to dashboard Cite

Web archives do not always capture every resource on every page that they attempt to archive. This results in archived pages missing a portion of their embedded resources. These embedded resources have varying historic, utility, and importance values. The proportion of missing embedded resources does not provide an accurate measure of their impact on the Web page; some embedded resources are more important to the utility of a page than others. We propose a method to measure the relative value of embedded resources and assign a damage rating to archived pages as a way to evaluate archival success. In this paper, we show that Web users' perceptions of damage are not accurately estimated by the proportion of missing embedded resources. In fact, the proportion of missing embedded resources is a less accurate estimate of resource damage than a random selection. We propose a damage rating algorithm that provides closer alignment to Web user perception, providing an overall improved agreement with users on memento damage by 17 % and an improvement by 51 % if the mementos have a damage rating delta >0.30. We use our algorithm to measure damage in the Internet Archive, showing that it is getting better at mitigating damage over time (going from a damage ). However, we show that a greater number of important embedded resources (2.05 per memento on average) are missing over time. Alternatively, the damage in WebCite is increasing over time (going from 0.375 in 2007 to 0.475 in 2014), while the missing embedded resources remain constant (13 % of the resources are missing on average). Finally, we investigate the impact of JavaScript on the damage of the archives, showing that a crawler that can archive JavaScript-dependent representations will reduce memento damage by 13.5 %.

show abstract

“…Ben Saad et al [13] claimed that using patterns is an effective way to predict changes, and then used this prediction to optimize the archiving process by crawling only important pages.…”

Section: Related Workmentioning

confidence: 99%

Detecting Off-Topic Pages in Web Archives

AlNoamany

Weigle

Nelson

2015

Research and Advanced Technology for Digital Libraries

View full text Add to dashboard Cite

Abstract. Web archives have become a significant repository of our recent history and cultural heritage. Archival integrity and accuracy is a precondition for future cultural research. Currently, there are no quantitative or content-based tools that allow archivists to judge the quality of the Web archive captures. In this paper, we address the problems of detecting off-topic pages in Web archive collections. We evaluate six different methods to detect when the page has gone off-topic through subsequent captures. Those predicted off-topic pages will be presented to the collection's curator for possible elimination from the collection or cessation of crawling. We created a gold standard data set from three ArchiveIt collections to evaluate the proposed methods at different thresholds. We found that combining cosine similarity at threshold 0.10 and change in size using word count at threshold −0.85 performs the best with accuracy = 0.987, F1 score = 0.906, and AUC = 0.968. We evaluated the performance of the proposed method on several Archive-It collections. The average precision of detecting the off-topic pages is 0.92.

show abstract

Archiving the web using page changes patterns: a case study

Cited by 18 publications

References 33 publications

Bots, Seeds and People

Bots, Seeds and People

Not all mementos are created equal: measuring the impact of missing resources

Detecting Off-Topic Pages in Web Archives

Contact Info

Product

Resources

About