A Constant Time Complexity Spam Detection Algorithm for Boosting Throughput on Rule-Based Filtering Systems

Abstract

Along with the barbarous growth of spams, anti-spam technologies including rule-based approaches and machine-learning thrive rapidly as well. In antispam industry, the rule-based systems (RBS) becomes the most prominent methods for fighting spam due to its capability to enrich and update rules remotely. However, the antispam filtering throughput is always a great challenge of RBS. Especially, the explosively spreading of obfuscated words leads to frequent rule update and extensive rule vocabulary expansion. These incremental obfuscated words make the filtering speed slow down and the throughput decrease. This paper addresses the challenging throughput issue and proposes a constant time complexity rule-based spam detection algorithm. The algorithm has a constant processing speed, which is independent of rule and its vocabulary size. A new special data structure, namely, Hash Forest, and a rule encoding method are developed to make constant time complexity possible. Instead of traversing each spam term in rules, the proposed algorithm manages to detect spam terms by checking a very small portion of all terms. The experiment results show effectiveness of proposed algorithm. INDEX TERMS Constant time complexity, hash forest, rule-based filtering, spam detection, throughput.

SpaML: a Bimodal Ensemble Learning Spam Detector based on NLP Techniques

“…They usually use dynamic Markov compression [20] and partial matching [21] to evaluate the model. Rule-based filtering systems [22] using behavioral methods or linguistic methods have also been proposed to score and classify texts. Deep Learning [23,24,25] is also beginning to be used to detect spam and algorithms such as CNN [26] and LSTM [27] are gaining ground.…”

Natural language processing: state of the art, current trends and challenges

“…It works using text categorization and in recent times, various machine learning techniques have been applied to text categorization or Anti-Spam Filtering like Rule Learning (Cohen 1996) [ 27 ], Naïve Bayes (Sahami et al, 1998; Androutsopoulos et al, 2000; Rennie.,2000) [ 5 , 109 , 115 ],Memory based Learning (Sakkiset al,2000b) [ 117 ], Support vector machines (Druker et al, 1999) [ 36 ], Decision Trees (Carreras and Marquez, 2001) [ 19 ], Maximum Entropy Model (Berger et al 1996) [ 14 ], Hash Forest and a rule encoding method (T. Xia, 2020) [ 153 ], sometimes combining different learners (Sakkis et al, 2001) [ 116 ]. Using these approaches is better as classifier is learned from training data rather than making by hand.…”

A Discrete Hidden Markov Model for SMS Spam Detection

“…However, throughput is a challenging issue of RBSs, and their time complexity of filtering algorithms could not be reduced to an acceptable level. To address the throughput issue, a constant time complexity spam detection algorithm was developed by Xia [20].…”