How much of the web is archived?

Ainsworth, Scott G.; AlSum, Ahmed; SalahEldeen, Hany M.; Weigle, Michele C.; Nelson, Michael L.

doi:10.1145/1998076.1998100

Cited by 81 publications

(88 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…First, researchers using web archive data have a subset of the full web. Using Ainsworth et al's (2013) estimates of web pages they might have between 35% and 90% of the web. By constructing their sample of URLs from DMOZ, Delicious, Bitly, and Google, Ainsworth et al (2013) almost certainly examined the inclusion of more popular and prominent URLs (i.e.…”

Section: Discussionmentioning

confidence: 99%

“…Using Ainsworth et al's (2013) estimates of web pages they might have between 35% and 90% of the web. By constructing their sample of URLs from DMOZ, Delicious, Bitly, and Google, Ainsworth et al (2013) almost certainly examined the inclusion of more popular and prominent URLs (i.e. the URLs included in DMOZ or added to Delicious are by definition more popular and prominent than the URLs that no one adds to these platforms).…”

Section: Discussionmentioning

confidence: 99%

“…The most complete study on the extent to which the web is archived is Ainsworth et al (2013). 1 They sampled 1,000 URLs each from the Open Directory Project (DMOZ), the recent URLs bookmarked on the social bookmarking site Delicious, randomly created hash values from Bitly, and the Google search engine index.…”

Section: Literaturementioning

confidence: 99%

“…counted a website as present in the archive as long as at least one page was archived. Ainsworth et al (2013) and Russell and Kane (2008) looked at web pages (URLs) from many websites but did not examine how much of each site was in the archive. We address this gap by analysing how much of a website has been archived and whether the archived pages in the website differ in a systematic way from the population of all pages on the website.…”

Section: Literaturementioning

confidence: 99%

“…While others have looked at archive coverage in terms of web pages (URLs) generally, notably Ainsworth et al (2013), this chapter is the first attempt to look at the extent of coverage of an individual website in depth. The remainder of this chapter is organized as follows.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

The Web as History : Using Web Archives to Understand the Past and the Present

2017

View full text Add to dashboard Cite

A CIP catalogue record for this book is available from The British Library.This book is published under a Creative Common 4.0 International license (CC BY 4.0). This license allows you to share, copy, distribute and transmit the work; to adapt the work and to make commercial use of the work providing attribution is made to the authors (but not in any way that suggests that they endorse you or your use of the work). Attribution should include the following information: Niels Brügger and Ralph Schroeder (eds. Table 5.2 Linear regression model explaining amount of country news mentions on BBC online 113 Table 5.3 Linear regression model explaining amount of country outlinks on BBC online 115 Table 6.1)Evaluation of the navigation and user interface of state websites 128 Table 7.1 Topics in three selected GeoCities neighbourhoods 149 Table 8.1 Direction and manifestation of ties in online networks 163 Table 8.2 Composition of sites (abortion stance) 167 Table 8.3 Composition of sites (site type) 167 Table 8.4 Top-20 sites ranked by Google, 2005 and 2015 169 Table 8.5 Network statistics 172 Table 8.6 Top-20 sites by indegree (full network) 176 Table 8.7 Top-20 sites by indegree (participant subnetwork) 178 Table 8.8 Top-20 sites by outdegree (full network) 179 Grant Blank is the Survey Research Fellow at the Oxford Internet Institute, University of Oxford. He is a sociologist specializing in the political and social impact of computers and the internet, the digital divide, statistical and qualitative methods, and cultural sociology. He is currently working on a project asking how cultural hierarchies are constructed in online reviews of cultural attractions. His other project links sample survey data with census data to generate small area estimates of Internet use in Great Britain. He holds a PhD from the University of Chicago. L iS t o f Co N t Ri B u to R S xivJonathan Bright is a Research Fellow at the Oxford Internet Institute, University of Oxford. He is a political scientist specialising in political communication and computational social science (especially 'big data' approaches to the social sciences). His research concerns how people get information about politics, and how this process is changing in the internet era.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Literaturementioning

confidence: 99%

Section: Literaturementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

The Web as History : Using Web Archives to Understand the Past and the Present

2017

View full text Add to dashboard Cite

show abstract

Format technology lifecycle analysis

Duretec

Becker

2017

Asso for Info Science & Tech

View full text Add to dashboard Cite

The lifecycles of format technology have been a defining concern for digital stewardship research and practice. However, little evidence exists to provide robust methods for assessing the state of any given format technology and describing its evolution over time.This article introduces relevant models from diffusion theory and market research and presents a replicable analysis method to compute models of technology evolution. Data cleansing and the combination of multiple data sources enable the application of non-linear regression to estimate the parameters of the Bass diffusion model on format technology market lifecycles.Through its application to a longitudinal dataset from the UK web archive, we demonstrate that the method produces reliable results and show that the Bass model can be used to describe format lifecycles. By analyzing adoption patterns across market segments, new insights are inferred about how the diffusion of formats and products such as applications occurs over time. The analysis provides a stepping stone to a more robust and evidence-based approach to model technology evolution.

show abstract

If these crawls could talk: Studying and documenting web archives provenance

Maemura

Worby

Milligan

et al. 2018

Asso for Info Science & Tech

View full text Add to dashboard Cite

The increasing use and prominence of web archives raises the urgency of establishing mechanisms for transparency in the making of web archives to facilitate the process of evaluating a web archive's provenance, scoping, and absences. Some choices and process events are captured automatically, but their interactions are not currently well understood or documented.This study examines the decision space of web archives and its role in shaping what is and what is not captured in the web archiving process. By comparing how three different web archives collections were created and documented, we investigate how curatorial decisions interact with technical and external factors and we compare commonalities and differences.The findings reveal the need to understand both the social and technical context that shapes those decisions and the ways in which these individual decisions interact. Based on the study, we propose a framework for documenting key dimensions of a collection that addresses the situated nature of the organizational context, technical specificities, and unique characteristics of web materials that are the focus of a collection. The framework enables future researchers to undertake empirical work studying the process of creating web archives collections in different contexts.

show abstract

How much of the web is archived?

Cited by 81 publications

References 18 publications

The Web as History : Using Web Archives to Understand the Past and the Present

The Web as History : Using Web Archives to Understand the Past and the Present

Format technology lifecycle analysis

If these crawls could talk: Studying and documenting web archives provenance

Contact Info

Product

Resources

About