Twelve Years of QBF Evaluations: QSAT Is PSPACE-Hard and It Shows

Abstract

Twelve years have elapsed since the first Quantified Boolean Formulas (QBFs) evaluation was held as an event linked to SAT conferences. During this period, researchers have striven to propose new algorithms and tools to solve challenging formulas, with evaluations periodically trying to assess the current state of the art. In this paper, we present an experimental account of solvers and formulas with the aim to understand the progress in the QBF arena across these years. Unlike typical evaluations, the analysis is not confined to the snapshot of submitted solvers and formulas, but rather we consider several tools that were proposed over the last decade, and we run them on different formulas from previous QBF evaluations. The main contributions of our analysis, which are also the messages we would like to pass along to the research community, are: (i) many formulas that turned out to be difficult to solve in past evaluations, remain still challenging after twelve years, (ii) there is no single solver which can significantly outperform all the others, unless specific categories of formulas are considered, and (iii) effectiveness of preprocessing depends both on the coupled solver and the structure of the formula.

Evaluating QBF Solvers: Quantifier Alternations Matter

“…We strengthen our above observations of performance diversity of solvers and solving paradigms with respect to numbers of qblocks by a virtual best solver (VBS) analysis, which is common in QBF [39] and SAT competitions (cf. [4]).…”

“…It is well known that preprocessing may have positive effects on the performance of certain solvers while negative effects on others (cf. [38,39]). To compensate for these effects, we applied preprocessing both to filter the original benchmark set S 17|523 and to preprocess instances.…”

“…In related work [39], the correlation between solver performance and various syntactic features such as treewidth [1,44] was analyzed. In contrast to that, we do not study such correlations.…”

DepQBF 6.0: A Search-Based QBF Solver Beyond Traditional QCDCL

“…We take preprocessing into account as it might have a positive impact on certain solvers while a negative on others (cf. [33,35]). Experiments were run on an AMD Opteron 6238 processor (2.6 GHz) under 64-bit Ubuntu Linux 12.04 with time and memory limits of 1800 seconds and seven GB.…”

QRATPre+: Effective QBF Preprocessing via Strong Redundancy Properties

“…The application of preprocessing prior to the actual solving process is crucial for the performance of most of the quantified Boolean formula (QBF) solvers [17,19,20]. Preprocessors aim at decreasing the complexity of a given formula with respect to the number of variables, the number of clauses, or the number of quantifier blocks.…”