We report on the benchmarks, participants and results of the third reactive synthesis competition (SYNTCOMP 2016). The benchmark library of SYNTCOMP 2016 has been extended to benchmarks in the new LTL-based temporal logic synthesis format (TLSF), and 2 new sets of benchmarks for the existing AIGER-based format for safety specifications. The participants of SYNTCOMP 2016 can be separated according to these two classes of specifications, and we give an overview of the 6 tools that entered the competition in the AIGER-based track, and the 3 participants that entered the TLSF-based track. We briefly describe the benchmark selection, evaluation scheme and the experimental setup of SYNTCOMP 2016. Finally, we present and analyze the results of our experimental evaluation, including a comparison to participants of previous competitions and a legacy tool.
IntroductionSince the definition of the problem more than 50 years ago [16], the automatic synthesis of reactive systems from formal specifications is one of the major challenges of computer science. Research into the basic questions related to the problem has led to a large body of theoretical results, but their impact on the practice of system design has been rather limited. To increase the impact of theoretical advancements in synthesis, the reactive synthesis competition (SYNTCOMP) has been founded in 2014 [27]. The competition is designed to foster research in scalable and user-friendly implementations of synthesis techniques by establishing a standard benchmark format, maintaining a challenging public benchmark library, and providing a dedicated and independent platform for the comparison of tools under consistent experimental conditions.The venues. A design choice for the first two competitions was to focus on safety properties specified as monitor circuits in an extension of the AIGER format known from the hardware model checking competition [15,25]. SYNTCOMP 2016 introduces the first major extension of the competition: in addition to the existing competition track, we introduce a new track that is based on properties in full linear temporal logic (LTL), given in the temporal logic synthesis format (TLSF) recently introduced by Jacobs, Klein and Schirmer [29].The organization team of SYNTCOMP 2016 consisted of R. Bloem and S. Jacobs, with technical assistance from J. Kreber for the setup and execution, and from F. Klein for the integration of TLSF.The rest of this paper describes the design, benchmarks, participants, and results of SYNTCOMP 2016. We present the benchmark set for SYNTCOMP 2016 in Section 2, followed by a description of the setup, rules and execution of the competition in Section 3. In Section 4 we give an overview of the participants of SYNTCOMP 2016, focusing on changes compared to last year's participants. Finally, the experimental results are presented and analyzed in Section 5.Note that more details on the goals and design of the competition can be found in the sister paper that discusses the design of SYNTCOMP in 2016 and the future [26].
Be...