OSN Crawling Schedulers and Their Implications onk-Plexes Detection

Abstract

Web crawlers are complex applications that explore the Web for different purposes. Web crawlers can be configured to crawl online social networks (OSNs) to obtain relevant data about their global structure. Before a web crawler can be launched to explore the Web, a large amount of settings have to be configured. These settings define the crawler's behavior and they have a big impact on the collected data. Both the amount of collected data and the quality of the information that it contains are affected by the crawler settings and, therefore, by properly configuring these web crawler settings we can target specific goals to achieve with our crawl. In this paper, we review the configuration choices that an attacker who wants to obtain information from an OSN by crawling it has to make to conduct his attack. We analyze different scheduler algorithms for web crawlers and evaluate their performance in terms of how useful they are to pursue a set of different adversary goals. C 2013 Wiley Periodicals, Inc.

Search engines crawling process optimization: a webserver approach

“…Their structure, however, has forced search engines to develop configurable crawlers to mitigate their crawling difficulties. Moreover, the amount and quality of the data collected are affected by crawlers' settings (Perez-Sola and Herrera-Joancomarti, 2013). Therefore, each type of these websites should be crawled by configured spiders behaving according to its structure.…”