A lexical approach for classifying malicious URLs

Abstract

Given the continuous growth of illicit activities on the Internet, there is a need for intelligent systems to identify malicious web pages. It has been shown that URL analysis is an e↵ective tool for detecting phishing, malware, and other attacks. Previous studies have performed URL classification using a combination of lexical features, network tra c, hosting information, and other strategies. These approaches require time-intensive lookups which introduce significant delay in real-time systems. This paper describes a lightweight approach for classifying malicious web pages using URL lexical analysis alone. The goal is to explore the upper-bound of the classification accuracy of a purely lexical approach. Another aim is to develop an approach which could be used in a real-time system. These goal culminate in the development of a classification system based on lexical analysis of URLs. It correctly classifies URLs of malicious web pages with 99.1% accuracy, a 0.4% false positive rate, an F1-Score of 98.7, and requires 0.62 milliseconds on average. This method substantially outperforms previously published algorithms on out-of-sample data.

SoK: A Comprehensive Reexamination of Phishing Research From the Security Perspective

“…of benign to malicious URLs, whereas in [28], [45], [49], [67], [77], [80] researchers evaluated their classifiers on highly unbalanced datasets (the benign dataset size is almost 50 times or more) but fixed ratios. Papers [35], [47], [49], [55], [69] report moderate benign-to-malicious data ratios (e.g., 2:1, 3:2, 5:1, etc.). Authors in [30], [43], [46], [54], [81]- [84] used slightly PhishTank (pub.)…”

“…c (pvt.) [21] [38], [72] [22], [41] [38], [41] [72], [77] a Gmail directory [34] b Spam URLs from UAB Spam DataMine email messages [38] c Private SpamTrap system [72], collected using blacklists and crowd-sourcing [41], [77] [78] a MalwareDomainslist [35] and malwareurl [48] b Malware private sources [78] unbalanced, or almost balanced, 9 data. Systems in [43], [59], [73] used more malicious URLs than legitimate ones.…”

“…In [35], authors proposed a lightweight phishing detection system using lexical features for link analysis. Such a system ensures separation between the victim and the attacker; and a lightweight scheme ensures speed and scalability.…”

“…Recall F-score 20 [30], [33]- [35], [47], [58]- [60], [69], [86] [22], [29], [50]- [52], [62]- [64], [70], [89] Error Rate 13 [21], [28], [34], [39], [40], [43], [49] [38], [65], [67], [73], [85], [89] Confusion Matrix a 20 [35], [43]- [46], [48], [57], [69], [76], [80], [82] [29], [50], [51], [58], [66], [78], [79], [88], [89] Area Under Curve 4 [33], [35], [50], [51] Others b 6 [42], [57],…”

“…b Balanced Success Rate [38], Root Mean Square Error [42], effective rules/patterns [54], Cost/Sum of classification [80], False Alarm rate [78] [30], [34], [41], [64], [77], [79] [21], [35], [44], [49]- [52], [59], [62] Bayesian Classifier 1 [34] Logistic Regression 11 [35], [43], [44], [47], [67], [79] [21], [28], [49], [63], [81] Naïve Bayes 11 [35], [43], [44], [52], [63], [77] [21], [30], [41], [49], [62] Support Vector Machines 12 [21], [40], [43], [49]- [51], [77] [28], [29], [41], [63], [66] RandomForest 14 [21], [41]<...>…”