Efficient Distributed Web Crawler Using Hefty and Enhanced Bandwidth Algorithms for Drug Website Search

Abstract

refabricate a proficient search structure is very important due to the current scale of the web. Search engines mine information from the web and a program called a web crawler, which efficiently surfs the web. A distributed crawler belongs to a variant of a web crawler, uses a dispersed computation method. In this paper, we design and implement the concept of Efficient Distributed Web Crawler using enhanced bandwidth and hefty algorithms. Mostly Web Crawler doesn't have any distributed cluster performance system and any implemented algorithm. In this paper, a novel Hefty Algorithm and enhanced bandwidth algorithm are combined for a better-distributed crawling system. The hefty algorithm was implemented to provide efficient and robust surfing results while applying on the drug web search. We also implemented the Enhanced Bandwidth algorithm to improve the efficiency of the proposed crawler.