A General Evaluation Framework for Topical Crawlers

Abstract

Topical crawlers are becoming important tools to support applications such as specialized Web portals, online searching, and competitive intelligence. As the Web mining field matures, the disparate crawling strategies proposed in the literature will have to be evaluated and compared on common tasks through well-defined performance measures. This paper presents a general framework to evaluate topical crawlers. We identify a class of tasks that model crawling applications of different nature and difficulty. We then introduce a set of performance measures for fair comparative evaluations of crawlers along several dimensions including generalized notions of precision, recall, and efficiency that are appropriate and practical for the Web. The framework relies on independent relevance judgements compiled by human editors and available from public directories. Two sources of evidence are proposed to assess crawled pages, capturing different relevance criteria. Finally we introduce a set of topic characterizations to analyze the variability in crawling effectiveness across topics. The proposed evaluation framework synthesizes a number of methodologies in the topical crawlers literature and many lessons learned from several studies conducted by our group. The general framework is described in detail and then illustrated in practice by a case study that evaluates four public crawling algorithms. We found that the proposed framework is effective at evaluating, comparing, differentiating and interpreting the performance of the four crawlers. For example, we found the IS crawler to be most sensitive to the popularity of topics.

A survey Paper on Web Crawler and their Implementation

“…The topic can consist of one or more seeds or even a short query. Preferential crawlers that start with only such information are often called topical crawlers [7,11,16]. They do not have text [3] classifiers to guide crawling.…”

“…They also use naïve Bayesian classifiers as a guide, but in this case the classifiers are trained to estimate the link distance between a crawled page and a set of relevant target pages [18]. A topical crawler [7,11,16], a MySpiders applet, is designed to demonstrate two topical crawling algorithms [1,7,11,16], best-N-first and InfoSpiders [10]. It is interactive in that a user submits a query and the Web is crawled in real time.…”

Population Control for Multi-agent Based Topical Crawlers

“…The study of the topology of the topical graphs in the Web has consequences on focused crawling and free exploration of Web pages or exploitation of already discovered resources [5]. Interesting works attempt to formulate the numerous variants involved in the design of this kind of system and to propose suited evaluation frameworks and metrics to research teams working in the field [6] [7].…”

Key Spatial Relations-based Focused Crawling (KSRs-FC) for Borderlands Situation Analysis

“…This method can make the formulization of a topic easier and intuitive, but it cannot express the semantic information of keywords. The second method uses an existing classified catalogue (such as Open Directory Project and Yahoo Directory) to define a given topic (Pant and Menczer, 2003;Srinivasan et al, 2005). The method could describe not only detailed information on topic itself but also some semantic information.…”