A focused crawler combinatory link and content model based on T-Graph principles

Abstract

Abstract-The two significant tasks of a focused Web crawler are finding relevant topic-specific documents on the Web and analytically prioritizing them for later effective and reliable download. For the first task, we propose a sophisticated custom algorithm to fetch and analyze the most effective HTML structural elements of the page as well as the topical boundary and anchor text of each unvisited link, based on which the topical focus of an unvisited page can be predicted and elicited with a high accuracy. Thus, our novel method uniquely combines both link-based and content-based approaches. For the second task, we propose a scoring function of the relevant URLs through the use of T-Graph (Treasure Graph) to assist in prioritizing the unvisited links that will later be put into the fetching queue. Our Web search system is called the Treasure-Crawler. This research paper embodies the architectural design of the Treasure-Crawler system which satisfies the principle requirements of a focused Web crawler, and asserts the correctness of the system structure including all its modules through illustrations and by the test results.

Empirical evaluation of the link and content-based focused Treasure-Crawler

“…1 illustrates the framework architecture of a Treasure-Crawler-based search engine and its modules which satisfy all the functional and non-functional requirements. The details of this architecture are giv-en in an under-review paper titled as: "A Focused Crawler Combinatory Link and Content Model Based on T-Graph Principles" [1]. These modules are designed in a way to have the ability of being plugged and played while requiring minimum changes in other modules or the adjacent module interfaces.…”

“…[8] In addition to the above procedure, the T-Graph structure as an exemplary guide carries out the task of priority association by providing a conceptual route for the crawler to follow and find on-topic regions. This phase is elaborated in [1].…”

“…o improve the quality of searching and indexing the Web, our proposed focused crawler depends on two main objectives, namely, to predict the topic of an unvisited page, and to prioritize the unvisited URLs within the current page by using a data structure called T-Graph. We elaborated the architecture of the Treasure-Crawler in a preceding paper [1], where a review on this subject field was discussed by naming and briefly describing some significant Web crawlers. Also, the requirements of a focused crawler were elicited and the evaluation criteria were outlined.…”

A Critique Empirical Evaluation of Relevance Computation for Focused Web Crawlers

“…Seyfi et al [11,12] proposed a focused crawler by using T-graph principles. This work gives solution to two problems in the focused crawler platform.…”