Overfitting in Synthesis: Theory and Practice

Abstract

In syntax-guided synthesis (SyGuS), a synthesizer's goal is to automatically generate a program belonging to a grammar of possible implementations that meets a logical specification. We investigate a common limitation across state-of-the-art SyGuS tools that perform counterexample-guided inductive synthesis (CEGIS). We empirically observe that as the expressiveness of the provided grammar increases, the performance of these tools degrades significantly.We claim that this degradation is not only due to a larger search space, but also due to overfitting. We formally define this phenomenon and prove no-free-lunch theorems for SyGuS, which reveal a fundamental tradeoff between synthesizer performance and grammar expressiveness.A standard approach to mitigate overfitting in machine learning is to run multiple learners with varying expressiveness in parallel. We demonstrate that this insight can immediately benefit existing SyGuS tools. We also propose a novel single-threaded technique called hybrid enumeration that interleaves different grammars and outperforms the winner of the SyGuS competition (Inv track), solving more problems and achieving a 5× mean speedup. *

Constraint-Based Relational Verification

“…Our pfwCSP solving technique builds on and generalizes a number of techniques developed for CHCs solving as well as invariant and ranking function discovery. Most closely related to our constraint solving method are CEGIS-based [51] and data-driven approaches to solving CHCs [14,22,24,26,27,38,44,45,48,49,65]. As remarked before, the new pfwCSP framework is strictly more expressive than CHCs and extending the prior techniques to the new framework is non-trivial.…”

“…But with very expressive templates like polyhedra, there could be many solutions, and a solution thus returned is liable to overfitting, i.e., the solution to the example instances becomes too specific to be an actual solution to the original input clauses. [44] discusses a similar overfitting issue in the context of grammar-based synthesis.…”

“…Our stratified CEGIS is inspired by the idea of stratified languages of predicates proposed in the context of predicate abstraction with CEGAR [34,55]. It is also similar in spirit to the work by Padhi et al [44], but they use a stratified family of grammars. Also none of these prior works use unsat cores for updating the language/grammar stratum, synthesize well-founded relations and functional predicates, or support non-Horn clauses.…”

“…The synthesis thus needs to use expressive templates without compromising the efficiency. Stratified CEGIS combines CEGIS [51] with stratified families of templates [55] (i.e., decomposing templates into a series of increasingly expressive templates) to achieve completeness in the sense of [34,55], a theoretical guarantee of convergence, and a faster and stable convergence by avoiding the overfitting problem of expressive templates to counterexamples [44]. The constraint solving method naturally generalizes a number of previous techniques developed for CHCs solving and invariant/ranking function synthesis, addressing the challenges due to the generality of pfwCSP that is essential for relational verification.…”

Grammar Filtering for Syntax-Guided Synthesis

“…Removing parts of a grammar is potentially dangerous though, as we may remove the possibility of finding a solution altogether. In fact, understanding the grammar's impact on synthesis algorithms is a complex problem, connected to the concept of overfitting (Padhi et al 2019).…”