Predicting rankings of software verification tools

Abstract

So ware veri cation competitions, such as the annual SV-COMP, evaluate so ware veri cation tools with respect to their e ectivity and e ciency. Typically, the outcome of a competition is a (possibly category-speci c) ranking of the tools. For many applications, such as building portfolio solvers, it would be desirable to have an idea of the (relative) performance of veri cation tools on a given veri cation task beforehand, i.e., prior to actually running all tools on the task.In this paper, we present a machine learning approach to predicting rankings of tools on veri cation tasks. e method builds upon so-called label ranking algorithms, which we complement with appropriate kernels providing a similarity measure for veri cation tasks. Our kernels employ a graph representation for so ware source code that mixes elements of control ow and program dependence graphs with abstract syntax trees. Using data sets from SV-COMP, we demonstrate our rank prediction technique to generalize well and achieve a rather high predictive accuracy. In particular, our method outperforms a recently proposed feature-based approach of Demyanova et al. (when applied to rank predictions). CCS CONCEPTS•Computing methodologies →Ranking; Support vector machines; Cross-validation; •So ware and its engineering →So ware veri cation; Formal so ware veri cation; KEYWORDSSo ware veri cation, machine learning, ranking.

PeSCo: Predicting Sequential Combinations of Verifiers

“…configurations. For this, we applied an extension of our rank prediction approach introduced in [7]. Basically, for a given verification task we predict an ordering of CPAchecker configurations, and then sequentially run these configurations.…”

“…By employing SVMs, we are able to choose a kernel function 2 (similar to Weisfeiler-Lehman kernels [12]) that is specifically designed for graph substructures. However, the function proposed in [7] needed to be computed between the input instance X (the graph of a verification task) and every training sample Y , which can be quite costly in practice. As a consequence, we have re-implemented this approach and now compute Weisfeiler-Lehman-based features of single graphs.…”

“…The prediction process starts by building an intermediate graph representation. This is followed by a feature extraction and the final ranking procedure (details in [7]). If a prediction is not achievable in a certain time frame, we fall back to the standard CPA-Seq.…”

Efficient Strategies for CEGAR-Based Model Checking

“…The advantage of using a diverse set of models is that we can identify the most suitable application areas. Furthermore, we compare lower lever parameters of CEGAR as opposed to most experiments in the literature [11,19,36,37], where different algorithms or tools are compared. We formulate and address a research question related to the effectiveness and efficiency of each of our contributions.…”

“…Experimental evaluation There are many works in the literature that focus on experimental evaluation and comparison of model checking algorithms [11,19,36,37]. However, they usually focus on a certain domain (e.g., SV-COMP).…”