Impact of URI Canonicalization on Memento Count

Abstract

Quantifying the captures of a URI over time is useful for researchers to identify the extent to which a Web page has been archived. Memento TimeMaps provide a format to list mementos (URI-Ms) for captures along with brief metadata, like Memento-Datetime, for each URI-M. However, when some URI-Ms are dereferenced, they simply provide a redirect to a different URI-M (instead of a unique representation at the datetime), often also present in the TimeMap. This infers that confidently obtaining an accurate count quantifying the number of non-forwarding captures for a URI-R is not possible using a TimeMap alone and that the magnitude of a TimeMap is not equivalent to the number of representations it identifies. In this work we discuss this particular phenomena in depth. We also perform a breakdown of the dynamics of counting mementos for a particular URI-R (google.com) and quantify the prevalence of the various canonicalization patterns that exacerbate attempts at counting using only a TimeMap. For google.com we found that 84.9% of the URI-Ms result in an HTTP redirect when dereferenced. We expand on and apply this metric to TimeMaps for seven other URI-Rs of large Web sites and thirteen academic institutions. Using a ratio metric DI for the number of URI-Ms without redirects to those requiring a redirect when dereferenced, five of the eight large web sites' and two of the thirteen academic institutions' TimeMaps had a ratio of ratio less than one, indicating that more than half of the URI-Ms in these TimeMaps result in redirects when dereferenced.

A Framework for Aggregating Private and Public Web Archives

“…In previous work [15], we highlighted that the URI-Ms in a TimeMap for google.com produce nearly 85% HTTP redirects when dereferenced. Determining how many mementos exist from an archive for a URI-R is thereby impossible from a TimeMap alone.…”

“…Additional information about a URI-M would be useful if present in a TimeMap. For example, knowing the HTTP status code of the dereferenced URI-M would reduce the amount of time needed to determine unique captures in the archive [15]. Extending TimeMaps may also provide the facility for the integration of private and public Web archives.…”

Aggregator Reuse and Extension for Richer Web Archive Interaction

“…Most research involving Memento aggregation relates to usage of the aggregator rather than enhancement of the aggregation process. In the same way that prior to MemGator, researchers would state "we requested URIs from the Time Travel Service", this statement was transformed to "we used MemGator to request URIs", indicative that it was useful for researchers to utilize their own aggregator instance [21,14,4]. A facet of this use case is the ability for researchers to customize the set of web archives to be used as the basis for querying, which is performed prior to running MemGator by modifying a configuration file 4 .…”

Profiling Web Archival Voids for Memento Routing

“…However, when we analyzed our data, we found that there were many fluctuations between "200" and "404", where some resources changed their status codes back and forth hundreds of times. It turned out that it was caused by lack of proper URI normalization/canonicalization [50], [51]. For example, when a TimeMap was requested for "apple.com" they returned "200", but for "Apple.com" or "APPLE.COM" they returned "404" instead.…”