Scalable, generic, and adaptive systems for focused crawling

Abstract

Focused crawling is the process of exploring a graph iteratively, focusing on parts of the graph relevant to a given topic. It occurs in many situations such as a company collecting data on competition, a journalist surfing the Web to investigate a political scandal, or an archivist recording the activity of influential Twitter users during a presidential election. In all these applications, users explore a graph (e.g., the Web or a social network), nodes are discovered one by one, the total number of exploration steps is constrained, some nodes are more valuable than others, and the objective is to maximize the total value of the crawled subgraph.In this article, we introduce scalable, generic, and adaptive systems for focused crawling. Our first effort is to define an abstraction of focused crawling applicable to a large domain of real-world scenarios. We then propose a generic algorithm, which allows us to identify and optimize the relevant subsystems. We prove the intractability of finding an optimal exploration, even when all the information is available.Taking this intractability into account, we investigate how the crawler can be steered in several experimental graphs. We show the good performance of a greedy strategy and the importance of being able to run at each step a new estimation of the crawling frontier. We then discuss this estimation through heuristics, selftrained regression, and multi-armed bandits. Finally, we investigate their scalability and efficiency in different real-world scenarios and by comparing with state-of-the-art systems.

Focused Crawling Through Reinforcement Learning

“…Then, a page from the host is taken using the online classifier. Similarly, Gouriten et al [12] use bandits to choose estimators for scoring the frontier.…”

A Frequent Named Entities-Based Approach for Interpreting Reputation in Twitter

“…Since we want to find the reputation of an entity and our goal is the richness and relevance of the sample, we find these methods not suitable. The idea of focused crawling related to a specific topic, based on weights, has been used in [12] and expert sampling in [10]. The study provided by [6] underlines the importance of the retweets and of the mentions in judging about the influence of the users.…”

Interpreting Reputation Through Frequent Named Entities in Twitter

“…The most important question that we should pose is: "Do we went a statistically representative sample that aligns with the real large Twitter dataset or do we want a filtered sample that focuses on the relevant tweets?". Several papers have contributed to find a statistically representative sample [6,8], while others highlight focused on crawling ( [9]) or Expert Sampling ( [7]). Inspired by the latest work, we consider three main parameters that influence the quality of the tweet: the number of times the tweet is retweeted, the favorite count of the tweet, and the number of followers of the user that has tweeted.…”