Online Web Bot Detection Using a Sequential Classification Approach

Cabri, Alberto; Suchacka, Grażyna; Rovetta, Stefano; Masulli, Francesco

doi:10.1109/hpcc/smartcity/dss.2018.00252

Cited by 16 publications

(13 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Web bot detection aims to accurately distinguish whether a web visitor is a bot or a human. This categorisation may entail simply distinguishing web bots from human visitors [3,4,10,12,18,34] or further categorising web bots based on their functionality [17], purpose [6,32,43], or complexity (i.e., simple vs. sophisticated) and based on whether they try to evade detection or not [20].…”

Section: Background and Related Workmentioning

confidence: 99%

“…The web bot detection approaches that are based on web logs rely primarily on several "traditional" machine learning algorithms, such as Support Vector Machines [20,29,37], Random Forests [20,34], Adaboost [20,34], and Multi-layer Perceptron classifiers [10,20,29,37]. Initially, the sessions of each visitor of the web server are extracted from the web logs [7,20,26,30,33,34].…”

Section: Background and Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Detection of Advanced Web Bots by Combining Web Logs with Mouse Behavioural Biometrics

et al. 2021

View full text Add to dashboard Cite

Web bots vary in sophistication based on their purpose, ranging from simple automated scripts to advanced web bots that have a browser fingerprint, support the main browser functionalities, and exhibit a humanlike behaviour. Advanced web bots are especially appealing to malicious web bot creators, due to their browser-like fingerprint and humanlike behaviour which reduce their detectability. This work proposes a web bot detection framework that comprises two detection modules: (i) a detection module that utilises web logs, and (ii) a detection module that leverages mouse movements. The framework combines the results of each module in a novel way to capture the different temporal characteristics of the web logs and the mouse movements, as well as the spatial characteristics of the mouse movements. We assess its effectiveness on web bots of two levels of evasiveness, (a) moderate web bots that have a browser fingerprint and (b) advanced web bots that have a browser fingerprint and also exhibit a humanlike behaviour. We show that combining web logs with visitors? mouse movements is more effective and robust towards detecting advanced web bots that try to evade detection, as opposed to using only one of those approaches.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Section: Background and Related Workmentioning

confidence: 99%

Detection of Advanced Web Bots by Combining Web Logs with Mouse Behavioural Biometrics

et al. 2021

View full text Add to dashboard Cite

show abstract

“…In both cases, a ground truth of human and Web bot sessions is required. In most recent research, the annotation process relies on comparing each visitor's agent name [26] and IP address [5,7,13,22,25] with the agent names and IPs of known web bots according to lists hosted on external servers. Such lists mostly contain identifiers for bots which are benign in nature, like, for example, search engine bots, although some malicious bots can be found there as well.…”

Section: Background and Related Workmentioning

confidence: 99%

“…To evaluate the framework, we split the dataset into two sets, one for training and one for testing. Our Web server by default splits the HTTP log data into files based on a log rotation technique 7 . The total files that were generated over a year were 13.…”

Section: Datasetmentioning

confidence: 99%

“…For our experiments, we incorporated 4 well known machine learning algorithms, all of whom have been used by other researchers for the Web bot detection problem. More specifically, we incorporated the Support Vector Machine [21,26], the Random Forest [25], the Adaboost [25] and the MLP classifiers [7,21,26]. Furthermore, we added an ensemble classifier, which we call the Voting classifier, that performs a class probability averaging of all the available classifiers [25].…”

Section: Classification Algorithms Testedmentioning

confidence: 99%

See 1 more Smart Citation

Towards a framework for detecting advanced Web bots

Iliou

Κωστούλας

Tsikrika

et al. 2019

Proceedings of the 14th International Conference on Availability, Reliability and Security

View full text Add to dashboard Cite

Automated programs (bots) are responsible for a large percentage of website traffic. These bots can either be used for benign purposes, such as Web indexing, Website monitoring (validation of hyperlinks and HTML code), feed fetching Web content and data extraction for commercial use or for malicious ones, including, but not limited to, content scraping, vulnerability scanning, account takeover, distributed denial of service attacks, marketing fraud, carding and spam. To ensure their security, Web servers try to identify bot sessions and apply special rules to them, such as throttling their requests or delivering different content. The methods currently used for the identification of bots are based either purely on rule-based bot detection techniques or a combination of rulebased and machine learning techniques. While current research has developed highly adequate methods for Web bot detection, these methods' adequacy when faced with Web bots that try to remain undetected hasn't been studied. For this reason, we created and evaluated a Web bot detection framework on its ability to detect conspicuous bots separately from its ability to detect advanced Web bots. We assessed the proposed framework performance using real HTTP traffic from a public Web server. Our experimental results show that the proposed framework has significant ability to detect Web bots that do not try to hide their bot identity using HTTP Web logs (balanced accuracy in a false-positive intolerant server > 95%). However, detecting advanced Web bots that present a browser fingerprint and may present a humanlike behaviour as well is considerably more difficult.

show abstract