The Dawn of Today's Popular Domains

Abstract

The Web has been around and maturing for 25 years. The popular websites of today have undergone vast changes during this period, with a few being there almost since the beginning and many new ones becoming popular over the years. This makes it worthwhile to take a look at how these sites have evolved and what they might tell us about the future of the Web. We therefore embarked on a longitudinal study spanning almost the whole period of the Web, based on data collected by the Internet Archive starting in 1996, to retrospectively analyze how the popular Web as of now has evolved over the past 18 years.For our study we focused on the German Web, specifically on the top 100 most popular websites in 17 categories. This paper presents a selection of the most interesting findings in terms of volume, size as well as age of the Web. While related work in the field of Web Dynamics has mainly focused on change rates and analyzed datasets spanning less than a year, we looked at the evolution of websites over 18 years. We found that around 70% of the pages we investigated are younger than a year, with an observed exponential growth in age as well as in size up to now. If this growth rate continues, the number of pages from the popular domains will almost double in the next two years. In addition, we give insights into our data set, provided by the Internet Archive, which hosts the largest and most complete Web archive as of today.

Modeling clusters from the ground up: A web data approach

“…6 Webpages that attract more traffic also have higher probability of being archived. Nevertheless, the consensus is that the IA is the most extensive and complete archive in the world (Ainsworth et al, 2011; Holzmann et al, 2016). Focusing on a subset of websites similar to the one used here, Thelwall and Vaughan (2004) indicate that the IA captures at least one webpage for 92% of all the US commercial websites.…”

The Past Web

“…An example of such a study is an analysis we conducted in 2016 of the dawn of today's most popular German domains over 18 years, i.e., the toplevel domain .de from 1996 to 2013, with the required data provided by the Internet Archive (Holzmann et al, 2016c). This investigation was carried out purely by studying metadata describing the archived records, without analysing actual payloads.…”

Collecting Primary Sources from Web Archives: A Tale of Scarcity and Abundance

“…The current prompt availability of a large variety of born-digital materials such as syllabi (Cohen, 2005), bachelor, master and doctoral theses (Ramage, 2011), academic websites (Holzmann et al, 2016b) and their hyperlinked structure (Hale et al, 2014) is about to become a new relevant component of this field of research (Nanni, 2017b).…”