Factors affecting website reconstruction from the web infrastructure

Abstract

When a website is suddenly lost without a backup, it may be reconstituted by probing web archives and search engine caches for missing content. In this paper we describe an experiment where we crawled and reconstructed 300 randomly selected websites on a weekly basis for 14 weeks. The reconstructions were performed using our web-repository crawler named Warrick which recovers missing resources from the Web Infrastructure (WI), the collective preservation effort of web archives and search engine caches. We examine several characteristics of the websites over time including birth rate, decay and age of resources. We evaluate the reconstructions when compared to the crawled sites and develop a statistical model for predicting reconstruction success from the WI. On average, we were able to recover 61% of each website's resources. We found that Google's PageRank, number of hops and resource age were the three most significant factors in determining if a resource would be recovered from the WI.

Page History Explorer: Visualizing and Comparing Page Histories

“…Directory page. Although, some solutions based on searching for similar content on the Web have been already proposed [6], [14], the problem of locating moved pages or their content parts is still unresolved.…”

“…McCown et al [14] measured the availability of page copies inside the repositories of major search engines and the Internet Archive. Their research was motivated by the need to provide efficient methods for reproducing the latest versions of Web sites.…”

“…Data could be also concurrently downloaded from other repositories [8], [14]. If there is error page encountered the system tries to re-download the snapshot for which error messages were returned instead.…”

“…Such a "journey to the past" may offer sentimental and informative values. Recently, multiple use cases of Web archives have been identified by the International Internet Preservation Consortium [7] and some researchers [13], [14]. For example, in one possible case, a local journalist wishes to quickly overview the content of a municipality homepage over a selected time period.…”

Why web sites are lost (and how they're sometimes found)

“…For example, Google's cache provided 95% of the resources for recovering website 2 but only 22% for website 1. More extensive experiments reconstructing 300 randomly selected websites over a period of three months have shown that on average 61% of a website's resources (77% textual, 42% images and 32% other) could be recovered if the website were lost and immediately reconstructed [7].…”