We need to talk about antiviruses: challenges &amp; pitfalls of AV evaluations

Botacin, Marcus; Ceschin, Fabrício; Geus, Paulo Lício de; Grégio, André

doi:10.1016/j.cose.2020.101859

Cited by 32 publications

(19 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A common, automated method for labeling malware is scanning with a single antivirus. However, antivirus signatures are frequently incorrect and can lack family information entirely [22,23]. Furthermore, antivirus vendors often use different names for the same malware family, which we refer to as aliases.…”

Section: Malware Dataset Labeling Strategiesmentioning

confidence: 99%

“…However, all approaches to malware grouping run into different versions of the same set of problems. Malware is written by an active adversary who attempts to evade and mislead analysts, including complex code obfuscations and misdirection via code theft to slow analysts and thwart automation [27,28,29,30]. Even standard countermeasures such as packing, which hides the original source code of a program from static analysis, are poorly understood and difficult to circumvent [31].…”

Section: Malware Dataset Labeling Strategiesmentioning

confidence: 99%

See 1 more Smart Citation

MOTIF: A Large Malware Reference Dataset with Ground Truth Family Labels

Joyce¹,

Amlani²,

Nicholas³

et al. 2021

Preprint

View full text Add to dashboard Cite

Malware family classification is a significant issue with public safety and research implications that has been hindered by the high cost of expert labels. The vast majority of corpora use noisy labeling approaches that obstruct definitive quantification of results and study of deeper interactions. In order to provide the data needed to advance further, we have created the Malware Open-source Threat Intelligence Family (MOTIF) dataset. MOTIF contains 3,095 malware samples from 454 families, making it the largest and most diverse public malware dataset with ground truth family labels to date, nearly 3× larger than any prior expert-labeled corpus and 36× larger than the prior Windows malware corpus. MOTIF also comes with a mapping from malware samples to threat reports published by reputable industry sources, which both validates the labels and opens new research opportunities in connecting opaque malware samples to human-readable descriptions. This enables important evaluations that are normally infeasible due to non-standardized reporting in industry. For example, we provide aliases of the different names used to describe the same malware family, allowing us to benchmark for the first time accuracy of existing tools when names are obtained from differing sources. Evaluation results obtained using the MOTIF dataset indicate that existing tasks have significant room for improvement, with accuracy of antivirus majority voting measured at only 62.10% and the well-known AVClass [1] tool having just 46.78% accuracy. Our findings indicate that malware family classification suffers a type of labeling noise unlike that studied in most ML literature, due to the large open set of classes that may not be known from the sample under consideration [2,

show abstract

Section: Malware Dataset Labeling Strategiesmentioning

confidence: 99%

Section: Malware Dataset Labeling Strategiesmentioning

confidence: 99%

MOTIF: A Large Malware Reference Dataset with Ground Truth Family Labels

Joyce¹,

Amlani²,

Nicholas³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Only recently have some of these AV biases been categorized and described. Botacin et al (2020) has shown that the detection rate of AV products may vary by country (i.e., is this malware global, or country specific, in its proliferation), executable type (e.g., COM files vs. DLL), and family type (e.g., ransomware vs trojans). These biases will naturally be embedded into any model and evaluation built from labels that are AV produced.…”

Section: Data Collection Challengesmentioning

confidence: 99%

“…These biases will naturally be embedded into any model and evaluation built from labels that are AV produced. Further, using older files to try and maximize confidence is not a guaranteed workaround, since AV engines will have label regressions over time, where they stop detecting sufficiently old files as malicious (Botacin et al, 2020).…”

Section: Data Collection Challengesmentioning

confidence: 99%

A Survey of Machine Learning Methods and Challenges for Windows Malware Classification

Raff,

Nicholas

2020

Preprint

View full text Add to dashboard Cite

Malware classification is a difficult problem, to which machine learning methods have been applied for decades. Yet progress has often been slow, in part due to a number of unique difficulties with the task that occur through all stages of the developing a machine learning system: data collection, labeling, feature creation and selection, model selection, and evaluation. In this survey we will review a number of the current methods and challenges related to malware classification, including data collection, feature extraction, and model construction, and evaluation. Our discussion will include thoughts on the constraints that must be considered for machine learning based solutions in this domain, and yet to be tackled problems for which machine learning could also provide a solution. This survey aims to be useful both to cybersecurity practitioners who wish to learn more about how machine learning can be applied to the malware problem, and to give data scientists the necessary background into the challenges in this uniquely complicated space.

show abstract

“…The global vs. local dichotomy is not a new observation. Previous studies have shown that other AV products can have different FP and false negative (FN) rates for regional malware (e.g., United States vs. Brazil) and malware types (e.g., Trojan vs. ransomware) [2]. This can cause excessive false alerts, which users find unacceptable and may lead them to abandon the malware detectors.…”

Section: Introductionmentioning

confidence: 99%

Getting Passive Aggressive About False Positives: Patching Deployed Malware Detectors

Raff¹,

Filar²,

Holt³

2020

Preprint

View full text Add to dashboard Cite

False positives (FPs) have been an issue of extreme importance for anti-virus (AV) systems for decades. As more security vendors turn to machine learning, alert deluge has hit critical mass with over 20% of all alerts resulting in FPs and, in some organizations, the number reaches half of all alerts [1]. This increase has resulted in fatigue, frustration, and, worst of all, neglect from security workers on SOC teams. A foundational cause for FPs is that vendors must build one global system to try and satisfy all customers, but have no method to adjust to individual local environments. This leads to outrageous, albeit technically correct, characterization of their platforms being 99.9% effective. Once these systems are deployed the idiosyncrasies of individual, local environments expose blind spots that lead to FPs and uncertainty.We propose a strategy for fixing false positives in production after a model has already been deployed. For too long the industry has tried to combat these problems with inefficient, and at times, dangerous allowlist techniques and excessive model retraining which is no longer enough. We propose using a technique called passive-aggressive learning to alter a malware detection model to an individual's environment, eliminating false positives without sharing any customer sensitive information. We will show how to use passive-aggressive learning to solve a collection of notoriously difficult false positives from a production environment without compromising the malware model's accuracy, reducing the total number of FP alerts by an average of 23x.

show abstract

We need to talk about antiviruses: challenges & pitfalls of AV evaluations

Cited by 32 publications

References 7 publications

MOTIF: A Large Malware Reference Dataset with Ground Truth Family Labels

MOTIF: A Large Malware Reference Dataset with Ground Truth Family Labels

A Survey of Machine Learning Methods and Challenges for Windows Malware Classification

Getting Passive Aggressive About False Positives: Patching Deployed Malware Detectors

Contact Info

Product

Resources

About