On the Lack of Consensus in Anti-Virus Decisions: Metrics and Insights on Building Ground Truths of Android Malware

Hurier, Médéric; Allix, Kevin; Bissyandé, Tegawendé F.; Klein, Jacques; Traon, Yves Le

doi:10.1007/978-3-319-40667-1_8

Cited by 37 publications

(41 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We further perform a correlation study on these malware to assess whether the number of vulnerabilities in an apk can be correlated to the number of AVs that flag it. This is important since AVs are known to lack consensus among themselves [82], [83]. For every vulnerability type we found that the Spearman's ρ was below 0.30, implying negligible correlation.…”

Section: Vulnerability and Malwarementioning

confidence: 83%

Understanding the Evolution of Android App Vulnerabilities

Gao

Kong

et al. 2021

IEEE Trans. Rel.

Self Cite

View full text Add to dashboard Cite

The Android ecosystem today is a growing universe of a few billion devices, hundreds of millions of users and millions of applications targeting a wide range of activities where sensitive information is collected and processed. Security of communication and privacy of data are thus of utmost importance in application development. Yet, regularly, there are reports of successful attacks targeting Android users. While some of those attacks exploit vulnerabilities in the Android OS, others directly concern application-level code written by a large pool of developers with varying experience. Recently, a number of studies have investigated this phenomenon, focusing however only on a specific vulnerability type appearing in apps, and based on only a snapshot of the situation at a given time. Thus, the community is still lacking comprehensive studies exploring how vulnerabilities have evolved over time, and how they evolve in a single app across developer updates. Our work fills this gap by leveraging a data stream of 5 million app packages to reconstruct versioned lineages of Android apps and finally obtained 28,564 app lineages (i.e., successive releases of the same Android apps) with more than 10 app versions each, corresponding to a total of 465,037 apks. Based on these app lineages, we apply stateof-the-art vulnerability-finding tools and investigate systematically the reports produced by each tool. In particular, we study which types of vulnerabilities are found, how they are introduced in the app code, where they are located, and whether they foreshadow malware. We provide insights based on the quantitative data as reported by the tools, but we further discuss the potential false positives. Our findings and study artifacts constitute a tangible knowledge to the community. It could be leveraged by developers to focus verification tasks, and by researchers to drive vulnerability discovery and repair research efforts.

show abstract

Section: Vulnerability and Malwarementioning

confidence: 83%

Understanding the Evolution of Android App Vulnerabilities

Gao

Kong

et al. 2021

IEEE Trans. Rel.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Hurier et al [23] define a set of metrics to characterize the discrepancy between malware detectors while treating the voting result as the ground truth, namely the problem (i) mentioned above. In contrast, we differentiate the voting result from the ground truth, and aim to quantify the distance between the voting result and the unknown ground truth.…”

Section: Related Workmentioning

confidence: 99%

Statistical Estimation of Malware Detection Metrics in the Absence of Ground Truth

Sun

Chen

et al. 2018

IEEE Trans.Inform.Forensic Secur.

View full text Add to dashboard Cite

The accurate measurement of security metrics is a critical research problem, because an improper or inaccurate measurement process can ruin the usefulness of the metrics. This is a highly challenging problem, particularly when the ground truth is unknown or noisy. In this paper, we measure five malware detection metrics in the absence of ground truth, which is a realistic setting that imposes many technical challenges. The ultimate goal is to develop principled, automated methods for measuring these metrics at the maximum accuracy possible. The problem naturally calls for investigations into statistical estimators by casting the measurement problem as a statistical estimation problem. We propose statistical estimators for these five malware detection metrics. By investigating the statistical properties of these estimators, we characterize when the estimators are accurate, and what adjustments can be made to improve them under what circumstances. We use synthetic data with known ground truth to validate these statistical estimators. Then, we employ these estimators to measure five metrics with respect to a large data set collected from VirusTotal.

show abstract

“…Second, the evaluation of systematic approaches is sensitive to some underlying assumptions which themselves are not grounded. For example, the choice of detection threshold and Anti-Virus engines can largely impact the characteristics of reference datasets [24]. Several flaws can also artificially improve the performance of detectors [2] or mislead the authors about the quality of their output [27].…”

Section: Related Workmentioning

confidence: 99%

“…Inconsistencies in Anti-Virus (AV) labels are indeed common. This is due to both naming disagreements [24], [25] across vendors and also a lack of adopted standards 1 for naming malware.…”

Section: Introductionmentioning

confidence: 99%

Euphony: Harmonious Unification of Cacophonous Anti-Virus Vendor Labels for Android Malware

Hurier

Suárez-Tangil

Dash

et al. 2017

2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR)

Self Cite

View full text Add to dashboard Cite

Android malware is now pervasive and evolving rapidly. Thousands of malware samples are discovered every day with new models of attacks. The growth of these threats has come hand in hand with the proliferation of collective repositories sharing the latest specimens. Having access to a large number of samples opens new research directions aiming at efficiently vetting apps. However, automatically inferring a reference dataset from those repositories is not straightforward and can inadvertently lead to unforeseen misconceptions. On the one hand, samples are often mis-labeled as different parties use distinct naming schemes for the same sample. On the other hand, samples are frequently mis-classified due to conceptual errors made during labeling processes. In this paper, we mine Anti-Virus labels and analyze the associations between all labels given by different vendors to systematically unify common samples into family groups. The key novelty of our approach, named EU-PHONY [20], is that no a-priori knowledge on malware families is needed. We evaluate EUPHONY using reference datasets and more than 400 thousands additional samples outside of these datasets. Results show that EUPHONY can accurately label malware with a fine-grained clustering of families, while providing competitive performance against the state-of-the-art.

show abstract

On the Lack of Consensus in Anti-Virus Decisions: Metrics and Insights on Building Ground Truths of Android Malware

Cited by 37 publications

References 22 publications

Understanding the Evolution of Android App Vulnerabilities

Understanding the Evolution of Android App Vulnerabilities

Statistical Estimation of Malware Detection Metrics in the Absence of Ground Truth

Euphony: Harmonious Unification of Cacophonous Anti-Virus Vendor Labels for Android Malware

Contact Info

Product

Resources

About