Optimal allocation of heterogeneous robots in world wide web search engines

Abstract

It is a conflicting requirement to reduce network traffic and loads on Web servers caused by the robots in a WorldWide Web search engine and to increase the currency of the database in the search engine. This paper addresses the problem of finding optimal allocations of a given number of robots in a search engine to heterogeneous Internet domains such that a cost function which characterizes the combined consideration of network/server loads and database currency can be minimized. The cost function and the optimization problem are formulated based on a queueing model of search engines. Our algorithm to solve the problem uses the depth-first search strategy to traverse an enumeration tree which generates all possible allocations.Indexing engine idling -When a robot bringing a new/changed page to the indexing engine finds the buffer empty, the indexing engine has been idle for a while. Therefore, the indexing engine is waiting for