Just-in-time learning for bottom-up enumerative synthesis

Abstract

A key challenge in program synthesis is the astronomical size of the search space the synthesizer has to explore. In response to this challenge, recent work proposed to guide synthesis using learned probabilistic models. Obtaining such a model, however, might be infeasible for a problem domain where no high-quality training data is available. In this work we introduce an alternative approach to guided program synthesis: instead of training a model ahead of time we show how to bootstrap one just in time, during synthesis, by learning from partial solutions encountered along the way. To make the best use of the model, we also propose a new program enumeration algorithm we dub guided bottom-up search, which extends the efficient bottom-up search with guidance from probabilistic models. We implement this approach in a tool called Probe, which targets problems in the popular syntax-guided synthesis (SyGuS) format. We evaluate Probe on benchmarks from the literature and show that it achieves significant performance gains both over unguided bottom-up search and over a state-of-the-art probability-guided synthesizer, which had been trained on a corpus of existing solutions. Moreover, we show that these performance gains do not come at the cost of solution quality: programs generated by Probe are only slightly more verbose than the shortest solutions and perform no unnecessary case-splitting.

Learning to Combine Per-Example Solutions for Neural Program Synthesis

“…The To provide fair comparison across all methods, each test split is run using a single core and single CPU thread with a timeout of 5s. To account for variability across machines, we chose to run a test split on a machine chosen randomly from a collection of 7 machines of similar configuration (Google Cloud instances with 120GB RAM each) 4 . We report standard error across the 30 test runs.…”

“…STUN [1] extends the CEGIS [28] approach by providing domain-specific explicit unification operators for combining partial solutions. Recently, BESTER [24] and later PROBE [4] perform bottom-up enumeration of programs in a loop by enumerating all programs that satisfy IO examples partially. This is followed by heuristics-based selection of promising programs.…”

The effect of distance metrics in a general purpose synthesizer of imperative programs: A second empirical study using enlarged search spaces

“…CONCORD uses deduction to constrain the search space and to guide the search, synthesising problems over lists using a functional programming language. Probe 29 (2020) is a synthesizer that trains a probabilistic model by learning from partial solutions encountered along the way, instead of training ahead of time. The authors also implement a new program enumeration algorithm, which extends the efficient bottom‐up search approach.…”

The effect of distance metrics in a general purpose synthesizer: An empirical study on integer domain imperative programs

“…The authors also implement a new program enumeration algorithm, which extends the efficient bottom-up search approach. Probe has been evaluated on 140 SyGuS benchmarks problems of string manipulation, bit-vector manipulation, and circuit transformation, outperforming state-of-art contemporary SyGuS synthesizers (such as EuPhony 18 and CVC4 19 ) in problems where larger train datasets are not available; 20 TerpreT-a model composed of a specification of a program representation (declarations of random variables) and an interpreter that describes how programs map inputs to outputs (a model connecting unknowns to observations); 21 and DeepCoder-a program synthesis technique from input-output examples guided by a neural network model. 22 Mandal et al 23 proposed a novel approach (named NetSyn) to learn the fitness function using neural networks for better guiding the process of searching for a solution.…”