An Empirical Methodology to Evaluate Vulnerability Discovery Models

Abstract

Vulnerability discovery models (VDMs) operate on known vulnerability data to estimate the total number of vulnerabilities that will be reported after a software is released. VDMs have been proposed by industry and academia, but there has been no systematic independent evaluation by researchers who are not model proponents. Moreover, the traditional evaluation methodology has some issues that biased previous studies in the field. In this work we propose an empirical methodology that systematically evaluates the performance of VDMs along two dimensions (quality and predictability) and addresses all identified issues of the traditional methodology. We conduct an experiment to evaluate most existing VDMs on popular web browsers' vulnerability data. Our comparison shows that the results obtained by the proposed methodology are more informative than those by the traditional methodology. Among evaluated VDMs, the simplest linear model is the most appropriate choice in terms of both quality and predictability for the first 6-12 months since a release date. Otherwise, logistics-based models are better choices.

A Screening Test for Disclosed Vulnerabilities in FOSS Components

“…When a vulnerability is fixed by only adding lines of code, there will be no evidence to track, and the authors in [20] conservatively assume that in such cases the whole version prior the fix (namely, code(r 0 )) is vulnerable. This screening test was appropriate for the empirical analysis of Vulnerability Discovery Models [39], which are typically based on the NVD and its cautious assumption "r 0 is vulnerable and so are all its previous versions" (see [20]), as this would create a consistent approximation of the NVD. Essentially, the overall approach can be seen as an instance of the generic screening test that we defined in Algorithm 1.…”

On the Security Cost of Using a Free and Open Source Component in a Proprietary Product

“…The simplest metric is time (since release), and the corresponding model is a Vulnerability Discovery Model. Massacci and Nguyen [14] provide a comprehensive survey and independent empirical validation of several vulnerability discovery models. Several other metrics have been used: code complexity metrics [25,24,16], developer activity metrics [24], static analysis defect densities [27], frequencies of occurrence of programming constructs [21,28], etc.…”

The sigmoidal growth of operating system security vulnerabilities: An empirical revisit

“…2 As a rule rather than an exception, the security patches from both Red Hat and Microsoft have covered multiple vulnerabilities. Although these counting issues are known to affect estimates (Massacci and Nguyen, 2014), the use of security advisories does not invalidate the theoretical premises as such; regardless whether the known vulnerabilities are observed individually or in groups, the resulting trends should follow a sigmoidal growth trend.…”

“…The literature contains an abundant amount of sigmoid functions for S-shaped growth patterns (Höök et al, 2011;López et al, 2004;Massacci and Nguyen, 2014;Meade and Islam, 2006;Wang et al, 2014;Zwietering et al, 1990). A classical example is the famous function that Gompertz (1825) formulated to determine the rate of mortality.…”

“…This theoretical background places the model into an interesting intersection of empirical software engineering research (Massacci and Nguyen, 2014). At the methodological front the model largely falls to the domain of software reliability modeling, but the theoretical argument is distinctively different.…”