Unsupervised Clustering of Web Sessions to Detect Malicious and Non-malicious Website Users

Analysis of Aggregated Bot and Human Traffic on E-Commerce Site

“…Consecutive user sessions were reconstructed based on the requests' timestamps, assuming a minimum 30-minute interval between two subsequent sessions of a given user (the value of 30 minutes has been commonly applied in previous Web traffic analyses, e.g. in [9], [19]). …”

“…requests with status codes starting with "4"). We decided to compute the aforementioned attributes because some previous user session analyses for non-ecommerce environments reported that these session features may be useful in distinguishing Web robots from human users [1], [8], [9].…”

“…Some previous studies reported that robots (especially crawlers) request mostly Web page files and ignore image files [8], [9], [24]. In contrast, human users navigate through the website following the structure of hyperlinks and when they open a new page, they usually download the page description file along with image files embedded in the page.…”

“…The most obvious ones are connected with potential threats of malicious bot activities [8], [9], [10]. Bot-generated click frauds in pay-per-click advertising result in higher fees paid by the advertisers [11].…”

“…Besides, bots consume network bandwidth and server resources; thus, they may cause degradation of the server performance and the quality of service offered to human users. Especially dangerous are automated DoS (Denial of Service) attacks, which may even make the server stall or crash [9]. Lastly, identification of robot traffic is essential when analyzing behavior of human users, who are characterized by different navigational patterns than bots [1], [8], [9], [12], [13], [14], [15].…”