Difficulties of Timestamping Archived Web Pages

Aturban, Mohamed; Nelson, Michael L.; Weigle, Michele C.

doi:10.48550/arxiv.1712.03140

Cited by 4 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also found that some URI-Ms redirected to mementos with memento-datetimes after 2016. Grusky et al did not encounter these problems in 2016 likely due to issues relating to web archive playback, as studied by Ainsworth et al [38] and Aturban et al [39], which we worked around by discarding mementos that redirected beyond 2016. After resolving these issues, shown in Table III, we were left with 277,724 mementos of news articles to evaluate.…”

Section: Methodsmentioning

confidence: 98%

It's All About The Cards: Sharing on Social Media Probably Encouraged HTML Metadata Growth

Jones,

Neblitt-Jones,

Weigle

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In a perfect world, all articles consistently contain sufficient metadata to describe the resource. We know this is not the reality, so we are motivated to investigate the evolution of the metadata that is present when authors and publishers supply their own. Because applying metadata takes time, we recognize that each news article author has a limited metadata budget with which to spend their time and effort. How are they spending this budget? What are the top metadata categories in use? How did they grow over time? What purpose do they serve? We also recognize that not all metadata fields are used equally. What is the growth of individual fields over time? Which fields experienced the fastest adoption? In this paper, we review 227,726 HTML news articles from 29 outlets captured by the Internet Archive between 1998 and 2016. Upon reviewing the metadata fields in each article, we discovered that 2010 began a metadata renaissance as publishers embraced metadata for improved search engine ranking, search engine tracking, social media tracking, and social media sharing. When analyzing individual fields, we find that one application of metadata stands out above all others: social cards -the cards generated by platforms like Twitter when one shares a URL. Once a metadata standard was established for cards in 2010, its fields were adopted by 20% of articles in the first year and reached more than 95% adoption by 2016. This rate of adoption surpasses efforts like Schema.org and Dublin Core by a fair margin. When confronted with these results on how news publishers spend their metadata budget, we must conclude that it is all about the cards.

show abstract

Section: Methodsmentioning

confidence: 98%

It's All About The Cards: Sharing on Social Media Probably Encouraged HTML Metadata Growth

Jones,

Neblitt-Jones,

Weigle

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Taking into account all of these archive-related issues, it becomes a challenging problem to distinguish between legitimate changes by archives and malicious changes. In our technical report [9] we provide several recommendations of how to generate repeatable xity information. Kuhn et al [32] de ne a trusty URI as a URI that contains a cryptographic hash value of the content it identi es as shown in Figure 2.…”

Section: Background and Related Workmentioning

confidence: 99%

“…(8) Name the le using its content-addressable hash. (9) Compress the block le to e ciently archive it. (10) Publish the compressed block le on a URI that contains its hash.…”

Section: Block Disseminationmentioning

confidence: 99%

“…based on the playback of archived resources (client-side) for two reasons. First, we are not expecting hashes generated and stored in WARC les by archives at crawl time to match those generated on the playback of mementos [9]. Second, if an archive has been compromised then it is likely the corresponding hashes have been also compromised, so we need to have the xity information stored in independent archives [36].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Archive Assisted Archival Fixity Verification Framework

Aturban

Alam

Nelson

et al. 2019

2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL)

Self Cite

View full text Add to dashboard Cite

e number of public and private web archives has increased, and we implicitly trust content delivered by these archives. Fixity is checked to ensure an archived resource has remained unaltered since the time it was captured. Some web archives do not allow users to access xity information and, more importantly, even if xity information is available, it is provided by the same archive from which the archived resources are requested. In this research, we propose two approaches, namely Atomic and Block, to establish and check xity of archived resources. In the Atomic approach, the xity information of each archived web page is stored in a JSON le (or a manifest), and published in a well-known web location (an Archival Fixity server) before it is disseminated to several on-demand web archives. In the Block approach, we rst batch together xity information of multiple archived pages in a single binary-searchable le (or a block) before it is published and disseminated to archives. In both approaches, the xity information is not obtained directly from archives. Instead, we compute the xity information (e.g., hash values) based on the playback of archived resources. One advantage of the Atomic approach is the ability to verify xity of archived pages even with the absence of the Archival Fixity server. e Block approach requires pushing fewer resources into archives, and it performs xity veri cation faster than the Atomic approach. On average, it takes about 1.25X, 4X, and 36X longer to disseminate a manifest to perma.cc, archive.org, and webcitation.org, respectively, than archive.is, while it takes 3.5X longer to disseminate a block to archive.org than perma.cc.e Block approach performs 4.46X faster than the Atomic approach on verifying the xity of archived pages. 3 wax.lib.harvard.edu/collections/home.do 4

show abstract

“…We describe the steps we took to create a data set of 16,627 mementos of 3,698 unique live web URIs (Uniform Resource Identifiers) from 17 public web archives. We use this collection in our study of identifying changes and transformations in the content of mementos over time (our preliminary work can be found in [6,7,8]).…”

Section: Introductionmentioning

confidence: 99%

Collecting 16K archived web pages from 17 public web archives

Aturban¹,

Nelson²,

Weigle³

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

We document the creation of a data set of 16,627 archived web pages, or mementos, of 3,698 unique live web URIs (Uniform Resource Identifiers) from 17 public web archives. We used four different methods to collect the dataset. First, we used the Los Alamos National Laboratory (LANL) Memento Aggregator to collect mementos of an initial set of URIs obtained from four sources: (a) the Moz Top 500, (b) the dataset used in our previous study, (c) the HTTP Archive, and (d) the Web Archives for Historical Research group. Second, we extracted URIs from the HTML of already collected mementos. These URIs were then used to look up mementos in LANL's aggregator. Third, we downloaded web archives' published lists of URIs of both original pages and their associated mementos. Fourth, we collected more mementos from archives that support the Memento protocol by requesting TimeMaps directly from archives, not through the Memento aggregator. Finally, we downsampled the collected mementos to 16,627 due to our constraints of a maximum of 1,600 mementos per archive and being able to download all mementos from each archive in less than 40 hours.

show abstract

Difficulties of Timestamping Archived Web Pages

Cited by 4 publications

References 12 publications

It's All About The Cards: Sharing on Social Media Probably Encouraged HTML Metadata Growth

It's All About The Cards: Sharing on Social Media Probably Encouraged HTML Metadata Growth

Archive Assisted Archival Fixity Verification Framework

Collecting 16K archived web pages from 17 public web archives

Contact Info

Product

Resources

About