Adversarial classification

Dalvi, Nilesh; Domingos, Pedro; Sanghai, Sumit; Verma, Deepak

doi:10.1145/1014052.1014066

Cited by 776 publications

(625 citation statements)

References 17 publications

Supporting

Mentioning

618

Contrasting

Unclassified

Order By: Relevance

“…However in the machine learning and pattern recognition literature the issue of the hardness of evasion in adversarial classification problems has not been deeply and formally investigated yet. Most of the works proposed countermeasures against specific kinds of attacks for spam filtering and intrusion detection tasks (see for instance [1][2][3]), and only few of them proposed formal models of adversarial classification tasks [4,5], or analysed the main issues raised by the application of machine learning techniques [6]. Therefore, from an engineering viewpoint the design of accurate and hard to evade classification systems for security applications is still an open problem.…”

Section: Introductionmentioning

confidence: 99%

Multiple Classifier Systems for Adversarial Classification Tasks

Biggio

Fumera

Roli

2009

Multiple Classifier Systems

View full text Add to dashboard Cite

Abstract. Pattern classification systems are currently used in security applications like intrusion detection in computer networks, spam filtering and biometric identity recognition. These are adversarial classification problems, since the classifier faces an intelligent adversary who adaptively modifies patterns (e.g., spam e-mails) to evade it. In these tasks the goal of a classifier is to attain both a high classification accuracy and a high hardness of evasion, but this issue has not been deeply investigated yet in the literature. We address it under the viewpoint of the choice of the architecture of a multiple classifier system. We propose a measure of the hardness of evasion of a classifier architecture, and give an analytical evaluation and comparison of an individual classifier and a classifier ensemble architecture. We finally report an experimental evaluation on a spam filtering task.

show abstract

Section: Introductionmentioning

confidence: 99%

Multiple Classifier Systems for Adversarial Classification Tasks

Biggio

Fumera

Roli

2009

Multiple Classifier Systems

View full text Add to dashboard Cite

show abstract

“…And even if the email spam problem were to be solved, it is not obvious that the solution would apply to spam in other media. The general problem of adversarial information filtering [44] -of which spam filtering is the prime example -is likely to be of interest for some time to come.…”

Section: The Spam Ecosystemmentioning

confidence: 99%

Email Spam Filtering: A Systematic Review

Cormack

2008

FNT in Information Retrieval

226

141

View full text Add to dashboard Cite

Spam is information crafted to be delivered to a large number of recipients, in spite of their wishes. A spam filter is an automated tool to recognize spam so as to prevent its delivery. The purposes of spam and spam filters are diametrically opposed: spam is effective if it evades filters, while a filter is effective if it recognizes spam. The circular nature of these definitions, along with their appeal to the intent of sender and recipient make them difficult to formalize. A typical email user has a working definition no more formal than "I know it when I see it." Yet, current spam filters are remarkably effective, more effective than might be expected given the level of uncertainty and debate over a formal definition of spam, more effective than might be expected given the state-of-the-art information retrieval and machine learning methods for seemingly similar problems. But are they effective enough? Which are better? How might they be improved? Will their effectiveness be compromised by more cleverly crafted spam?We survey current and proposed spam filtering techniques with particular emphasis on how well they work. Our primary focus is spam filtering in email; Similarities and differences with spam filtering in other communication and storage media -such as instant messaging and the Web -are addressed peripherally. In doing so we examine the definition of spam, the user's information requirements and the role of the spam filter as one component of a large and complex information universe. Well-known methods are detailed sufficiently to make the exposition self-contained, however, the focus is on considerations unique to spam. Comparisons, wherever possible, use common evaluation measures, and control for differences in experimental setup. Such comparisons are not easy, as benchmarks, measures, and methods for evaluating spam filters are still evolving. We survey these efforts, their results and their limitations. In spite of recent advances in evaluation methodology, many uncertainties (including widely held but unsubstantiated beliefs) remain as to the effectiveness of spam filtering techniques and as to the validity of spam filter evaluation methods. We outline several uncertainties and propose experimental methods to address them.

show abstract

“…For example, within the Probably Approximately Correct framework, Kearns and Li bound the classification error an adversary can cause with control over a fraction of the training set [10]. Dalvi et al apply game theory to the classification problem [6]. They model the interactions between the classifier and attacker as a game and develop an optimal counter-strategy for an optimal classifier playing against an optimal opponent.…”

Section: Related Workmentioning

confidence: 99%

Misleading Learners: Co-opting Your Spam Filter

Nelson

Barreno

Jack

et al. 2009

Machine Learning in Cyber Trust

124

View full text Add to dashboard Cite

Using statistical machine learning for making security decisions introduces new vulnerabilities in large scale systems. We show how an adversary can exploit statistical machine learning, as used in the SpamBayes spam filter, to render it useless-even if the adversary's access is limited to only 1% of the spam training messages. We demonstrate three new attacks that successfully make the filter unusable, prevent victims from receiving specific email messages, and cause spam emails to arrive in the victim's inbox. IntroductionApplications use statistical machine learning to perform a growing number of critical tasks in virtually all areas of computing. The key strength of machine learning is adaptability; however, this can become a weakness when an adversary manipulates the learner's environment. With the continual growth of malicious activity and electronic crime, the increasingly broad adoption of learning makes assessing the vulnerability of learning systems to attack an essential problem.The question of robust decision making in systems that rely on machine learning is of interest in its own right. But for security practitioners, it is especially important, as a wide swath of security-sensitive applications build on machine learning technology, including intrusion detection systems, virus and worm detection systems, and spam filters [13,14,18,20,24].Past machine learning research has often proceeded under the assumption that learning systems are provided with training data drawn from a natural distribution of inputs. However, in many real applications an attacker might have the ability to provide a machine learning system with maliciously chosen inputs that cause the system to infer poor classification rules. In the spam domain, for example, the adversary can send carefully crafted spam messages 1 Comp. Sci. Div., Soda Hall #1776, University of California, Berkeley, 94720-1776, USA 17and Reliability, DOI: 10.1007/978-0-387-88735-7_2, In Machine Learning in Cyber Trust: Security, Privacy, Reliability, eds. J. Tsai and P..Yu (eds.) Springer, 2009, pp. 17-51 18 that a human user will correctly identify and mark as spam, but which can influence the underlying machine learning system and adversely affect its ability to correctly classify future messages.We demonstrate how attackers can exploit machine learning to subvert the SpamBayes statistical spam filter. Our attack strategies exhibit two key differences from previous work: traditional attacks modify attack instances to evade a filter, whereas our attacks interfere with the training process of the learning algorithm and modify the filter itself; and rather than focusing only on placing spam emails in the victim's inbox, we also present attacks that remove legitimate emails from the inbox.We consider attackers with one of two goals: expose the victim to an advertisement or prevent the victim from seeing a legitimate message. Potential revenue gain for a spammer drives the first goal, while the second goal is motivated, for example, by an organiza...

show abstract

Adversarial classification

Cited by 776 publications

References 17 publications

Multiple Classifier Systems for Adversarial Classification Tasks

Multiple Classifier Systems for Adversarial Classification Tasks

Email Spam Filtering: A Systematic Review

Misleading Learners: Co-opting Your Spam Filter

Contact Info

Product

Resources

About