Accurate Malware Detection by Extreme Abstraction

When Malware is Packin' Heat; Limits of Machine Learning Classifiers Based on Static Analysis Features

“…We showed that machine-learning-based anti-malware engines on VirusTotal produce a substantial number of false positives on packed binaries, which can be due to the limitations discussed in this work. This is especially a serious issue for machine-learning-based approaches that frequently rely on labels from VirusTotal [22,86,88,97], causing an endless loop in which new approaches rely on polluted datasets, and, in turn, generate polluted datasets for future work.…”

“…As these numbers show, any approach that fails to consider packed benign samples when designing and evaluating a malware detection approach ultimately results in a substantial number of false positives on real-world data. This is especially a concern for machine-learning-based approaches, which, in the absence of reliable and fresh ground truth, frequently rely on labels from anti-malware products available on VirusTotal [22,43,86,88,97]. Given the disagreement of anti-malware products in labeling samples [42,48,65,99], a common practice is to sanitize a dataset, for example, by considering decisions from a selected set of anti-malware products, or, as another example, by using a voting-based consensus.…”

“…Packed benign samples that are detected by anti-malware products as malicious are incorrectly used as malware samples. For example, a recent related work [22] used a similar procedure for labeling, as stated by the authors: "We train a classifier using supervised learning and therefore require a target label for each sample (0 for benign and 1 for malware). We use malware indicators from VirusTotal.…”

“…Therefore, it is no longer acceptable for anti-malware products to detect anything packed as malicious. RQ4 is important because most machine-learning-based approaches rely on labels from VirusTotal in the absence of a reliable and fresh ground-truth dataset [22,86,88,97]. To this end, we identified six products on VirusTotal that, either on the corresponding company's website or on a VirusTotal blog post, are described as machine-learning-based malware detectors that use only static analysis features.…”

“…As packing is being increasingly adopted by legitimate software [84], the anti-malware industry needs to do better than detecting packers, otherwise good and bad programs are misclassified, causing pain to users and eventually resulting in alert fatigue and missed detections. This is especially a concern for previous studies that rely on anti-malware products for establishing ground truth, as misclassification of packed benign programs might have biased those studies [22,43,86,88,97].…”