A Fast Distributed Focused-web Crawling

Achsan, Harry Tursulistyono Yani; Wibowo, Wahyu Catur

doi:10.1016/j.proeng.2014.03.017

Cited by 34 publications

(19 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Distributed focused crawling is implemented using C# in Ref . C# provides libraries to access data via computer networks and to develop multithreading architecture.…”

Section: Current Status Of Web Crawlermentioning

confidence: 99%

A survey of Web crawlers for information retrieval

Kumar

Bhatia

Rattan

2017

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Performance of any search engine relies heavily on its Web crawler. Web crawlers are the programs that get webpages from the Web by following hyperlinks. These webpages are indexed by a search engine and can be retrieved by a user query. In the area of Web crawling, we still lack an exhaustive study that covers all crawling techniques. This study follows the guidelines of systematic literature review and applies it to the field of Web crawling. We used the standard procedure of carrying out a systematic literature review on 248 studies from a total of 1488 articles published in 12 leading journals and other premier conferences and workshops. Existing literature about the Web crawler is classified into different key subareas. Each subarea is further divided according to the techniques being used. We analyzed the distribution of various articles using multiple criteria and depicted conclusions. Various studies that use open source Web crawlers are also reported. We have highlighted future areas of research. We call for an increased awareness in various fields of the Web crawler and identify how techniques from other domains can be used for crawling the Web. Limitations and recommendations for future are also discussed. WIREs Data Mining Knowl Discov 2017, 7:e1218. doi: 10.1002/widm.1218 This article is categorized under: Algorithmic Development > Web Mining Fundamental Concepts of Data and Knowledge > Information Repositories Fundamental Concepts of Data and Knowledge > Motivation and Emergence of Data Mining

show abstract

“…Distributed focused crawling is implemented using C# in Ref . C# provides libraries to access data via computer networks and to develop multithreading architecture.…”

Section: Current Status Of Web Crawlermentioning

confidence: 99%

A survey of Web crawlers for information retrieval

Kumar

Bhatia

Rattan

2017

WIREs Data Min & Knowl

View full text Add to dashboard Cite

show abstract

“…It can be developed by: a) web log analysis using Association Rule [61], fuzzy classification [62], and fuzzy clustering [63,64], b) semantic enrichment on twitter messages and on emails, and c) tags analysis of social media. Since expertise is specialized skills and knowledge [64], it can be expressed as knowledge representation.…”

Section: Expertisementioning

confidence: 99%

“…Scopus, a bibliographic database, hold more than 55 million records [61]. The largest bibliographic database is WorldCat that holds 2 billion items [62].…”

Section: Sourcesmentioning

confidence: 99%

Ontology Enrichment for Multi-Domain Knowledge and Expertise Representation

Achsan¹,

Suhartanto²,

Wibowo³

2017

DAAAM Proceedings

Self Cite

View full text Add to dashboard Cite

Expertise is specialized skills and knowledge. It can be shared or disseminated if stated explicitly using expertise representation. Since people have multi area/domain expertise, we need multi-domain expertise representation. Someone's expertise can be recognized by his/her publication. In this article we proposed a new expertise representation by utilizing bibliography (publication metadata) and annotating ontology (ontology enrichment). The expertise representation is multi-domain that can be integrated semantically as easy as single-domain ontology.

show abstract

“…When the information needs are specific, possibly requiring in-depth research, there is a need to build niche search engines. There is also growing interest in knowledge discovery systems that typically rely on topical collections [13][14][15]. A collection of Web pages is said to form a topical collection when it is primarily focused on a topic or theme.…”

Section: Search Enginesmentioning

confidence: 99%

A Frame Work for Topical Collections Make with Focused and Accelerated Focused Crawlers

Saturi¹,

Raju²,

Kumar³

et al. 2015

IJCA

View full text Add to dashboard Cite

The rapid growth of the World-Wide Web poses unprecedented scaling challenges for general-purpose crawlers and search engines. In the personalized search domain, an alternative to general purpose crawler called focused crawlers are receiving increasing attention. The goal of these crawlers is to selectively seek out pages that are relevant to a pre-defined set of topics or theme. Rather than collecting and indexing all accessible Web documents to be able to answer all possible ad-hoc queries, these crawlers analyzes their crawl boundary to find the links that are likely to be most relevant for the crawl, and avoids irrelevant regions of the Web. This leads to significant savings in hardware and network resources, and helps keep the crawl more up-to-date. This paper presents and compares two focused crawlers called traditional focused crawler and accelerated focused crawler. Accelerated focused crawler takes offline lessons from traditional focused crawler. It emulates human surfer by trying to predict the relevance of a "HREF" target page based on words around the link on the source page. The topics are specified using exemplary documents in these experiments. Naive Bayesian classifier is used to guide the crawlers. The crawlers were evaluated for different number of pages crawled, for different number of features gathered from different distances from the link and with different feature selection methods.

show abstract

A Fast Distributed Focused-web Crawling

Cited by 34 publications

References 9 publications

A survey of Web crawlers for information retrieval

A survey of Web crawlers for information retrieval

Ontology Enrichment for Multi-Domain Knowledge and Expertise Representation

A Frame Work for Topical Collections Make with Focused and Accelerated Focused Crawlers

Contact Info

Product

Resources

About