Focused crawling: a new approach to topic-specific Web resource discovery

Abstract

The rapid growth of the World-Wide Web poses unprecedented scaling challenges for general-purpose crawlers and search engines. In this paper we describe a new hypertext resource discovery system called a Focused Crawler. The goal of a focused crawler is to selectively seek out pages that are relevant to a pre-defined set of topics. The topics are specified not using keywords, but using exemplary documents. Rather than collecting and indexing all accessible Web documents to be able to answer all possible ad-hoc queries, a focused crawler analyzes its crawl boundary to find the links that are likely to be most relevant for the crawl, and avoids irrelevant regions of the Web. This leads to significant savings in hardware and network resources, and helps keep the crawl more up-to-date.To achieve such goal-directed crawling, we designed two hypertext mining programs that guide our crawler: a classifier that evaluates the relevance of a hypertext document with respect to the focus topics, and a distiller that identifies hypertext nodes that are great access points to many relevant pages within a few links. We report on extensive focused-crawling experiments using several topics at different levels of specificity. Focused crawling acquires relevant pages steadily while standard crawling quickly loses its way, even though they are started from the same root set. Focused crawling is robust against large perturbations in the starting set of URLs. It discovers largely overlapping sets of resources in spite of these perturbations. It is also capable of exploring out and discovering valuable resources that are dozens of links away from the start set, while carefully pruning the millions of pages that may lie within this same radius. Our anecdotes suggest that focused crawling is very effective for building high-quality collections of Web documents on specific topics, using modest desktop hardware.

A Method for Focused Crawling Using Combination of Link Structure and Content Similarity

“…The primary metric which was used to evaluate the performance of the crawling system was the harvest rate P(C), which is the percentage of the web pages crawled which are related to the domain. Most focused crawlers have used this metric [6], [7], [4]. The core improvement of our focused crawler derives from combining link structure analysis and content similarity.…”

TS-IDS Algorithm for Query Selection in the Deep Web Crawling

“…The challenges include locating the data sources [16,17,18,19], learning and understanding the interface and the returned results so that query submission and data extraction can be automated [20,19,21,22].…”

Mining for Information Discovery on the Web: Overview and Illustrative Research

“…A much more efficient way is to crawl intelligently to retrieve only Web pages that are likely to be about machine learning. Such focused crawling (also known as topic distillation) has been the subject of much recent research [12,71,91]. The crawler can be guided by supervised learning techniques, in form of a Web document classifier [12,91], or by reinforcement learning techniques [71].…”

“…Such focused crawling (also known as topic distillation) has been the subject of much recent research [12,71,91]. The crawler can be guided by supervised learning techniques, in form of a Web document classifier [12,91], or by reinforcement learning techniques [71]. A well-known example of a taxonomy constructed from such focused crawling was the machine learning portal at cora.justresearch.com.…”