On establishing a benchmark for evaluating static analysis alert prioritization and classification techniques

Abstract

Benchmarks provide an experimental basis for evaluating software engineering processes or techniques in an objective and repeatable manner. We present the FAULTBENCH benchmark, as a contribution to current benchmark materials, for evaluation and comparison of techniques that prioritize and classify alerts generated by static analysis tools. Alert prioritization and classification addresses the problem in many static analysis tools of numerous alerts that are not an indication of a fault or unimportant to the developer. We utilized FAULTBENCH to evaluate three versions of the AWARE adaptive ranking model to prioritize and classify static analysis alerts. Individual FAULTBENCH subjects have different best prioritization and classification techniques. Using a single subject to evaluate a prioritization and classification technique could provide incorrect results. Together, FAULTBENCH subjects provide a precise and general evaluation of alert prioritization and classification techniques.

On the "naturalness" of buggy code

“…Our approach ranks warnings on a different aspect of source code than those they consider and could be used to complement their model. Finally, Heckman et al proposed Faultbench, a benchmark for comparison and evaluation of static analysis alert prioritization and classification techniques [18] and used it to validate the Aware [19] tool to prioritize static analysis tool warnings. Since results of our approach are promising, further research could investigate our approach against this additional benchmark.…”

Evaluating the relation between coding standard violations and faultswithin and across software versions

“…Zitser [6]. Heckman et al proposed a benchmark and procedures for the evaluation of software inspection prioritization and classification techniques [11]. Unfortunately, the benchmark is focused at Java programs.…”

Finding patterns in static analysis alerts: improving actionable alert ranking

“…We use three different classification algorithms to classify the feature vectors: decision tree (ADTree), naive Bayes and Bayesian network (BayesNet). The selection of these three classifiers is based on our experience classifying alerts from the FaultBench v0.1 [17,13] benchmark. For all classifiers we use the default parameters.…”

“…To do this we use the FaultBench v0.3 [16] method proposed by Heckman and Williams [13]. This technique uses the source code history of a project to determine if alerts are actionable or unactionable.…”