A review on techniques for optimizing web crawler results

Lawankar, Anuja; Mangrulkar, Nikhil

doi:10.1109/startup.2016.7583952

Cited by 6 publications

(1 citation statement)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The term crawler comes from the first search engine: the "WebCrawler". Since their first appearance, many efforts have been devoted to increasing the crawling procedure's efficiency and effectiveness [2,18,33]. Focused web crawling [10] is one of the main strategies to improve the crawling quality in specific contexts.…”

Section: Crawlingmentioning

confidence: 99%

On the feasibility of crawling-based attacks against recommender systems1

Aiolli

Conti

Picek

et al. 2022

JCS

View full text Add to dashboard Cite

Nowadays, online services, like e-commerce or streaming services, provide a personalized user experience through recommender systems. Recommender systems are built upon a vast amount of data about users/items acquired by the services. Such knowledge represents an invaluable resource. However, commonly, part of this knowledge is public and can be easily accessed via the Internet. Unfortunately, that same knowledge can be leveraged by competitors or malicious users. The literature offers a large number of works concerning attacks on recommender systems, but most of them assume that the attacker can easily access the full rating matrix. In practice, this is never the case. The only way to access the rating matrix is by gathering the ratings (e.g., reviews) by crawling the service’s website. Crawling a website has a cost in terms of time and resources. What is more, the targeted website can employ defensive measures to detect automatic scraping. In this paper, we assess the impact of a series of attacks on recommender systems. Our analysis aims to set up the most realistic scenarios considering both the possibilities and the potential attacker’s limitations. In particular, we assess the impact of different crawling approaches when attacking a recommendation service. From the collected information, we mount various profile injection attacks. We measure the value of the collected knowledge through the identification of the most similar user/item. Our empirical results show that while crawling can indeed bring knowledge to the attacker (up to 65% of neighborhood reconstruction on a mid-size dataset and up to 90% on a small-size dataset), this will not be enough to mount a successful shilling attack in practice.

show abstract

Section: Crawlingmentioning

confidence: 99%

On the feasibility of crawling-based attacks against recommender systems1

Aiolli

Conti

Picek

et al. 2022

JCS

View full text Add to dashboard Cite

show abstract

A novel user trend‐based priority assigner and URL scheduler for dynamic incremental crawling

Gupta

Dixit

2021

Concurrency and Computation

View full text Add to dashboard Cite

An efficient search engine needs to be designed in such a way that is able to provide relevant and accurate information in accordance with user needs and interests. The quality of downloaded records can be guaranteed only when website pages of high pertinence are downloaded by the crawlers in accordance with the current topics or user trends. Earlier Focused Crawlers were used to download topic specific pages but these crawlers were not able to adapt to the changing interest of the users. Therefore, there is a need to design crawlers that are able to naturally track the present pattern points and download site pages that meet client's present need. In this paper, a priority assigner and scheduler method for organizing Uniform Resource Locators (URLs) is being proposed that helps the crawler in tracking user's interest and prioritize downloading documents that are relevant to the user's choice in addition to current trends. The experimental results conforms that the proposed priority assigner and URL scheduler-based crawling outshines conventional crawling strategies based on Change-history or Site-Map-based methods in terms of quality of downloaded web pages and reducing network traffic over the Internet.

show abstract