Feature evaluation for web crawler detection with data mining techniques

Stevanovic, Dusan; An, Aijun; Vlajic, Natalija

doi:10.1016/j.eswa.2012.01.210

Cited by 70 publications

(50 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Another system represented in this regard is converting malicious requests to signature by using anomaly extension [3]. Another system with the name of MINDS extended after [4]. MINDS was detecting different kinds of intrusion in network layers with different elements such as scan detector, anomaly detector and summarization components.…”

Section: Related Workmentioning

confidence: 99%

Intrusion Detection System Based on Web Usage Mining

Mabzool¹,

Lighvan²

2014

IJCSEA

View full text Add to dashboard Cite

This artical present a system developed to find cyber threats automatically based on web usage mining methods in application layer. This system is an off-line intrusion detection system which includes different part to detect attacks and as a result helps find different kinds of attacks with different dispersals. In this study web server access logs used as the input data and after pre-processing, scanners and all identified attacks will be detected. As the next step, vectors feature from web access logs and parameters sent by HTTP will derived by three different means and at the end by employment of two clustering algorithms based on K-Means, anomaly behaviour of data are detached. Tentative results derived from this system represent that used methods are more applicable than similar systems because this system covers different kinds of attacks and mostly increase the accuracy and decrease false alarms.

show abstract

Section: Related Workmentioning

confidence: 99%

Intrusion Detection System Based on Web Usage Mining

Mabzool¹,

Lighvan²

2014

IJCSEA

View full text Add to dashboard Cite

show abstract

“…Although using the traditional search engine algorithm, technology and tool can acquire the expected network information, its efficiency and accuracy are limited because the search is based on key word. Thus, web crawler technology, which can acquire the web information based on the customers' needs and search the entire network, has become the research focus [9][10][11] . Based on the existing research results, this paper discusses and analyzes the path planning, error control, strategy implementation of the search as well the realization of the system.…”

Section: Introductionmentioning

confidence: 99%

Network Information Target Search Model and Strategy Based on Web Vertical Crawler System

Wang¹,

Jian²,

Wang³

et al. 2016

IJFGCN

View full text Add to dashboard Cite

show abstract

“…Although other features such as the standard deviation of requested page depth, the percentage of consecutive sequential http requests, the html-to-image ratio, and the percentage of 4xx error responses are robot session features used in previous machine learning classifiers [13], they may not be indicative of the future resource request behavior of Web robots. This is because the kinds of resources requested by a Web robot are likely to be independent of session level features.…”

Section: Classification Algorithmsmentioning

confidence: 99%

“…Stevanovic et al argue that two features, namely the standard deviation of requested pagedepth and the percentage of consecutive sequential HTTP requests belonging to the same web directory, are essential to separate robots from humans in Web server logs [13]. Yang et al also consider association rule mining to create an n-gram model of occurrence frequencies.…”

Section: Related Workmentioning

confidence: 99%

Request type prediction for Web robot and Internet of Things traffic

Rude¹

2015

Preprint

View full text Add to dashboard Cite

Abstract-The volume of Web robot traffic seen by Web servers and clouds continue to increase with the popularity of Internet of Things (IoT) devices. Such traffic exhibits decidedly different statistical and resource request patterns compared to humans. However, the optimizations ensuring high levels of Web systems and cloud performance requires traffic to exhibit the statistical and behavioral patterns of humans, not robots. This necessitates the design of novel Web system optimizations to handle Web robot traffic effectively. Caches are a basic component of high performing Web systems, but their effectiveness relies on accurate resource request prediction. In this paper, we explore a suite of classifiers for the resource request type prediction problem for robot traffic. Our analysis reveals: (i) a striking difference in the request patterns of robots across multiple servers from the same domain; and (ii) that Elman neural networks hold promise to predict request types despite these differences.

show abstract

Feature evaluation for web crawler detection with data mining techniques

Cited by 70 publications

References 13 publications

Intrusion Detection System Based on Web Usage Mining

Intrusion Detection System Based on Web Usage Mining

Network Information Target Search Model and Strategy Based on Web Vertical Crawler System

Request type prediction for Web robot and Internet of Things traffic

Contact Info

Product

Resources

About