Combining text and link analysis for focused crawling—An application for vertical search engines

Almpanidis, George; Kotropoulos, Constantine; Pitas, Ioannis

doi:10.1016/j.is.2006.09.004

Cited by 75 publications

(48 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Customized search engines can be categorized into general-purpose and topic-specific search engines [29]. Customized general-purpose search engines are equipped with a web crawler, which can continually collect large numbers of webpages by starting from a series of seed URLs (a list of URLs that the crawler starts with) and without a specific topic [32,41,42].…”

Section: Surface Geospatial Web Services Discoverymentioning

confidence: 99%

Discovering Land Cover Web Map Services from the Deep Web with JavaScript Invocation Rules

Hou

Chen

2016

IJGI

View full text Add to dashboard Cite

Automatic discovery of isolated land cover web map services (LCWMSs) can potentially help in sharing land cover data. Currently, various search engine-based and crawler-based approaches have been developed for finding services dispersed throughout the surface web. In fact, with the prevalence of geospatial web applications, a considerable number of LCWMSs are hidden in JavaScript code, which belongs to the deep web. However, discovering LCWMSs from JavaScript code remains an open challenge. This paper aims to solve this challenge by proposing a focused deep web crawler for finding more LCWMSs from deep web JavaScript code and the surface web. First, the names of a group of JavaScript links are abstracted as initial judgements. Through name matching, these judgements are utilized to judge whether or not the fetched webpages contain predefined JavaScript links that may prompt JavaScript code to invoke WMSs. Secondly, some JavaScript invocation functions and URL formats for WMS are summarized as JavaScript invocation rules from prior knowledge of how WMSs are employed and coded in JavaScript. These invocation rules are used to identify the JavaScript code for extracting candidate WMSs through rule matching. The above two operations are incorporated into a traditional focused crawling strategy situated between the tasks of fetching webpages and parsing webpages. Thirdly, LCWMSs are selected by matching services with a set of land cover keywords. Moreover, a search engine for LCWMSs is implemented that uses the focused deep web crawler to retrieve and integrate the LCWMSs it discovers. In the first experiment, eight online geospatial web applications serve as seed URLs (Uniform Resource Locators) and crawling scopes; the proposed crawler addresses only the JavaScript code in these eight applications. All 32 available WMSs hidden in JavaScript code were found using the proposed crawler, while not one WMS was discovered through the focused crawler-based approach. This result shows that the proposed crawler has the ability to discover WMSs hidden in JavaScript code. The second experiment uses 4842 seed URLs updated daily. The crawler found a total of 17,874 available WMSs, of which 11,901 were LCWMSs. Our approach discovered a greater number of services than those found using previous approaches. It indicates that the proposed crawler has a large advantage in discovering LCWMSs from the surface web and from JavaScript code. Furthermore, a simple case study demonstrates that the designed LCWMS search engine represents an important step towards realizing land cover information integration for global mapping and monitoring purposes.

show abstract

Section: Surface Geospatial Web Services Discoverymentioning

confidence: 99%

Discovering Land Cover Web Map Services from the Deep Web with JavaScript Invocation Rules

Hou

Chen

2016

IJGI

View full text Add to dashboard Cite

show abstract

“…The main challenges in focused crawling relate to the prioritization of URLs not yet visited, which may be based on similarity measures [24,26], hyperlink distance-based limits [30,31], or combinations of text and hyperlink analysis with Latent Semantic Indexing (LSI) [32]. Machine learning approaches, including naïve Bayes classifiers [25,33], Hidden Markov Models [34], reinforcement learning [35], genetic algorithms [36], and neural networks [37], have also been applied to prioritize the unvisited URLs.…”

Section: Focused and Deep-web Crawlingmentioning

confidence: 99%

ARCOMEM Crawling Architecture

Plachouras¹,

Carpentier

Faheem

et al. 2014

Future Internet

View full text Add to dashboard Cite

The World Wide Web is the largest information repository available today. However, this information is very volatile and Web archiving is essential to preserve it for the future. Existing approaches to Web archiving are based on simple definitions of the scope of Web pages to crawl and are limited to basic interactions with Web servers. The aim of the ARCOMEM project is to overcome these limitations and to provide flexible, adaptive and intelligent content acquisition, relying on social media to create topical Web archives. In this article, we focus on ARCOMEM's crawling architecture. We introduce the overall architecture and we describe its modules, such as the online analysis module, which computes a priority for the Web pages to be crawled, and the Application-Aware Helper which takes into account the type of Web sites and applications to extract structure from crawled content. We also describe a large-scale distributed crawler that has been developed, as well as the modifications we have implemented to adapt Heritrix, an open source crawler, to the needs of the project. Our experimental results from real crawls show that ARCOMEM's crawling architecture is effective in acquiring focused information about a topic and leveraging the information from social media. Future Internet2014, 6 519

show abstract

“…The availability of a huge volume of data through Internet has lead to an increasing interest in knowledge retrieval from large amount of hypertext (Almpanidis et al 2007). Two methods for extracting links among websites are available today: crawlers and SEs.…”

Section: Data Extractionmentioning

confidence: 99%

A study of global and local visibility as web indicators of research production

Torres

Díaz‐Fernández

2013

Research Evaluation

View full text Add to dashboard Cite

The concept of webpage visibility is usually linked to search engine optimization, and it is based on global in-link metric, that is, the number of received links from other websites, but without considering the sources of these links. The purpose of this article is to demonstrate that this global idea of visibility is only weakly correlated with web metrics measured over a network of related institutions or organizations (local visibility) and research production. As a case study, global and local visibility measurements have been obtained for a set of Spanish Universities, and they have been correlated with results provided by international rankings like the Webometrics Ranking of World's Universities and the Academic Ranking of World Universities by Shanghai Jiao Tung University. Obtained results suggest that the development of web indicators to be included as part of Universities evaluation programs should consider a local idea of visibility, considering a certain geographical context or similar related institutions.

show abstract

Combining text and link analysis for focused crawling—An application for vertical search engines

Cited by 75 publications

References 41 publications

Discovering Land Cover Web Map Services from the Deep Web with JavaScript Invocation Rules

Discovering Land Cover Web Map Services from the Deep Web with JavaScript Invocation Rules

ARCOMEM Crawling Architecture

A study of global and local visibility as web indicators of research production

Contact Info

Product

Resources

About