Scraping SERPs for Archival Seeds

Abstract

Event-based collections are often started with a web search, but the search results you find on Day 1 may not be the same as those you find on Day 7. In this paper 1 , we consider collections that originate from extracting URIs (Uniform Resource Identifiers) from Search Engine Result Pages (SERPs). Specifically, we seek to provide insight about the retrievability of URIs of news stories found on Google, and to answer two main questions: first, can one "refind" the same URI of a news story (for the same query) from Google after a given time? Second, what is the probability of finding a story on Google over a given period of time?To answer these questions, we issued seven queries to Google every day for over seven months (2017-05-25 to 2018-01-12) and collected links from the first five SERPs to generate seven collections for each query. The queries represent public interest stories: "healthcare bill," "manchester bombing, " "london terrorism, " "trump russia, " "travel ban, " "hurricane harvey," and "hurricane irma." We tracked each URI in all collections over time to estimate the discoverability of URIs from the first five SERPs. Our results showed that the daily average rate at which stories were replaced on the default Google SERP ranged from 0.21 -0.54, and a weekly rate of 0.39 -0.79, suggesting the fast replacement of older stories by newer stories. The probability of finding the same URI of a news story after one day from the initial appearance on the SERP ranged from 0.34 -0.44. After a week, the probability of finding the same news stories diminishes rapidly to 0.01 -0.11. In addition to the reporting of these probabilities, we also provide two predictive models for estimating the probability of finding the URI of an arbitrary news story on SERPs as a function of time. The web archiving community considers link rot and content drift important reasons for collection building. Similarly, our findings suggest that due to the difficulty in retrieving the URIs of news stories from Google, collection building that originates from search engines should begin as soon as possible in order to capture the first stages of events, and should persist in order to capture the evolution of the events, because it becomes more difficult to find the same news stories with the same queries on Google, as time progresses.

Web Tarayıcılarında Tohum URL Seçimi ve Performans Analizi: Kapsamlı Bir İnceleme

“…Ayrıca tohum URL seçiminde en çok kullanılan yöntemler; manuel seçim [7][8][9], DMOZ ve curlie.org [10,11] gibi açık kaynak dizinlerinden yapılan seçim ve Twitter [12,13] gibi sosyal medyadaki kullanıcıların paylaştıkları URL'ler üzerinden seçimlerdir. Bunlara ek özellikle odaklı tarayıcılarda Google ve Yahoo gibi arama motorları ile yapılan aramalarda, ortaya çıkan URL'leri, tohum URL olarak seçen çalışmalarda mevcuttur [14][15][16][17].…”

Creating Structure in Web Archives with Collections: Different Concepts from Web Archivists

“…Ogden et al [34] studied web archivists themselves to better understand the ways in which they "shape and maintain the preserved Web." Nwala et al [33] analyzed how to leverage search engine results to populate web archive collections. Nwala et al [32] also used Archive-It collections to compare human-made vs. automatically or semi-automatically generated collections.…”

The Past Web

“…Can we build event collections from web archives? Even if resources about events remain on the live Web, Nwala et al (Nwala et al, 2018b) detailed how they become more challenging to discover via search engine results as we get farther from the event. Topical focused crawling provides resources whose terms closely match the terms of a desired topic, such as an event, and these crawlers stop when the matching score for new content is too low.…”