How Well Are Arabic Websites Archived?

Abstract

It is has long been anecdotally known that web archives and search engines favor Western and English-language sites. In this paper we quantitatively explore how well indexed and archived are Arabic language web sites. We began by sampling 15,092 unique URIs from three different website directories: DMOZ (multi-lingual), Raddadi and Star28 (both primarily Arabic language). Using language identification tools we eliminated pages not in the Arabic language (e.g., English language versions of Al-Jazeera sites) and culled the collection to 7,976 definitely Arabic language web pages. We then used these 7,976 pages and crawled the live web and web archives to produce a collection of 300,646 Arabic language pages. We discovered: 1) 46% are not archived and 31% are not indexed by Google (www.google.com), 2) only 14.84% of the URIs had an Arabic country code top-level domain (e.g., .sa) and only 10.53% had a GeoIP in an Arabic country, 3) having either only an Arabic GeoIP or only an Arabic top-level domain appears to negatively impact archiving, 4) most of the archived pages are near the top level of the site and deeper links into the site are not wellarchived, 5) the presence in a directory positively impacts indexing and presence in the DMOZ directory, specifically, positively impacts archiving.

The Past Web

“…In previous works on web dynamics, suitable datasets had to be crawled first, which is tedious and can only be done for shorter periods (Cho and Garcia-Molina, 2000;Fetterly et al, 2003;Koehler, 2002;Adar et al, 2009). With access to existing archives, more recent studies of the Web were conducted retrospectively on available data (Hale et al, 2014;Agata et al, 2014;Alkwai et al, 2015), commonly with a focus on a particular subset, such as national domains or topical subsets. These kinds of works are typical data-centric tasks as they require access to archived raw data or metadata records.…”

“…Ainsworth et al (Ainsworth et al, 2011) investigated how much of the web was archived and estimated that 35 -90% of existing web resources have at least one Memento. Alkwai et al estimated the archive coverage of Arabic websites (Alkwai et al, 2015), and later conducted an additional study (Alkwai et al, 2017) to compare the archiving rates of English-, Arabic-, Danishand Korean-language web pages. Alkwai showed that English has a higher archiving rate than Arabic, which in turn has a higher archiving rate than Danish or Korean.…”

Profiling Web Archival Voids for Memento Routing

“…They reported that about 15% to 31% (depending on the sample) URIs are archived at least once per month. Alkwai et al revisited the archival rate question in 2015, but for web pages of specific languages [17], [18]. They collected over 15,000 URI samples from English, Arabic, Danish, and Korean languages to find out how much of the pages from each of these languages are archived.…”

Accessing web archives from different perspectives with potential synergies

“…With access to existing Web archives, more recent studies of the Web were conducted retrospectively on available data [29,30,31]. However, instead of analyzing the whole archive at once, all of them focus on a particular subset, such as national domains.…”