Untitled

Crawlers are deployed by a Web search engine for collecting information from different Web servers in order to maintain the currency of its data base of Web pages. We present studies on the optimization of Web search engines from different perspectives. We first investigate the number of crawlers to be used by a search engine so as to maximize the currency of the data base without putting an unnecessary load on the network. Both the static setting, where crawlers are always active, and the dynamic setting where, crawlers may be activated/deactivated as a function of the state of the system, are addressed. We then consider the optimal scheduling of the visits of these crawlers to the Web pages assuming these pages are modified at different rates. Finally, we briefly discuss some other optimization issues of Web search engines, including page ranking and system optimization.

show abstract

Optimal allocation of heterogeneous robots in world wide web search engines

Proceedings 15th International Parallel and Distributed Processing Symposium. IPDPS 2001

View full text Add to dashboard Cite

It is a conflicting requirement to reduce network traffic and loads on Web servers caused by the robots in a WorldWide Web search engine and to increase the currency of the database in the search engine. This paper addresses the problem of finding optimal allocations of a given number of robots in a search engine to heterogeneous Internet domains such that a cost function which characterizes the combined consideration of network/server loads and database currency can be minimized. The cost function and the optimization problem are formulated based on a queueing model of search engines. Our algorithm to solve the problem uses the depth-first search strategy to traverse an enumeration tree which generates all possible allocations.Indexing engine idling -When a robot bringing a new/changed page to the indexing engine finds the buffer empty, the indexing engine has been idle for a while. Therefore, the indexing engine is waiting for

show abstract

Untitled

Cited by 7 publications

References 3 publications

Indexing and retrieval for the Web

Indexing and retrieval for the Web

Optimization Issues in Web Search Engines

Optimal allocation of heterogeneous robots in world wide web search engines

Contact Info

Product

Resources

About