Web Crawler: A Review

Kausar, Md. Abu; Dhaka, Vijaypal Singh; Singh, Sanjeev Kumar

doi:10.5120/10440-5125

Cited by 86 publications

(40 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Ref represents a survey of domain‐specific query forms that is related only to hidden Web. Ref also represents a review of crawler but it is short. To carry out this research, we framed four research questions to determine the current status of Web crawlers.…”

Section: Discussionmentioning

confidence: 99%

A survey of Web crawlers for information retrieval

Kumar

Bhatia

Rattan

2017

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Performance of any search engine relies heavily on its Web crawler. Web crawlers are the programs that get webpages from the Web by following hyperlinks. These webpages are indexed by a search engine and can be retrieved by a user query. In the area of Web crawling, we still lack an exhaustive study that covers all crawling techniques. This study follows the guidelines of systematic literature review and applies it to the field of Web crawling. We used the standard procedure of carrying out a systematic literature review on 248 studies from a total of 1488 articles published in 12 leading journals and other premier conferences and workshops. Existing literature about the Web crawler is classified into different key subareas. Each subarea is further divided according to the techniques being used. We analyzed the distribution of various articles using multiple criteria and depicted conclusions. Various studies that use open source Web crawlers are also reported. We have highlighted future areas of research. We call for an increased awareness in various fields of the Web crawler and identify how techniques from other domains can be used for crawling the Web. Limitations and recommendations for future are also discussed. WIREs Data Mining Knowl Discov 2017, 7:e1218. doi: 10.1002/widm.1218 This article is categorized under: Algorithmic Development > Web Mining Fundamental Concepts of Data and Knowledge > Information Repositories Fundamental Concepts of Data and Knowledge > Motivation and Emergence of Data Mining

show abstract

Section: Discussionmentioning

confidence: 99%

A survey of Web crawlers for information retrieval

Kumar

Bhatia

Rattan

2017

WIREs Data Min & Knowl

View full text Add to dashboard Cite

show abstract

“…A web crawler (also known as a spider) is "a software or a programmed script that browses the WWW in a systematic, automated manner" [58], and systematically downloads numerous webpages starting from a seed URL [9]. Web crawlers date back to the 1990s, where they were introduced when the WWW was invented.…”

Section: Web Crawlers and Crawling Techniquesmentioning

confidence: 99%

“…Later on, web crawlers that could efficiently download millions of webpages were built. We can consider the Internet to be a "directed graph" where each node represents a webpage, and the edges represent hyperlinks connecting these webpages [58]. Web crawlers traverse over this graph-like structure of the Internet, go to webpages, and download their content for indexing purposes.…”

Section: Web Crawlers and Crawling Techniquesmentioning

confidence: 99%

Change Detection and Notification of Web Pages

et al. 2020

View full text Add to dashboard Cite

The majority of currently available webpages are dynamic in nature and are changing frequently. New content gets added to webpages, and existing content gets updated or deleted. Hence, people find it useful to be alert for changes in webpages that contain information that is of value to them. In the current context, keeping track of these webpages and getting alerts about different changes have become significantly challenging. Change Detection and Notification (CDN) systems were introduced to automate this monitoring process, and to notify users when changes occur in webpages. This survey classifies and analyzes different aspects of CDN systems and different techniques used for each aspect. Furthermore, the survey highlights the current challenges and areas of improvement present within the field of research.

show abstract

“…RESPONSE TIME This is the time taken for each query search to be completed by the engines; it can be measured using stop clock or as is displayed by some search engines Kausar, Dhaka and Singh (2013) RELATIVE RECALL This is the ability of a system to retrieve all or most of the relevant documents in the collection. The relative recall can be calculated using the following formulae;…”

Section: Precisionmentioning

confidence: 99%

Evaluating search effectiveness of some selected search engines

Bute¹,

Hussaini²,

Adamu³

2017

Glo Jnl Pure Appl Sci

View full text Add to dashboard Cite

With advancement in technology, many individuals are getting familiar with the internet a lot of users seek for information on the World Wide Web (WWW) using variety of search engines. This research work evaluates the retrieval effectiveness of Google, Yahoo, Bing, AOL and Baidu. Precision, relative recall and response time were considered for this evaluation, a total of 24 search queries were sampled based on information queries, navigational queries and transactional queries. They are categorized into single word queries, double word queries, sentence queries and alphanumeric queries. Finally the overall average of all the designated queries shows Bing has the best score based on relative recall with 0.54, Google has the best response time with 0.39s and has the highest precision with 1.37.

show abstract

Web Crawler: A Review

Cited by 86 publications

References 22 publications

A survey of Web crawlers for information retrieval

A survey of Web crawlers for information retrieval

Change Detection and Notification of Web Pages

Evaluating search effectiveness of some selected search engines

Contact Info

Product

Resources

About