Question selection for interactive program synthesis

Ji, Ruyi; Liang, Jingjing; Xiong, Yingfei; Zhang, Lu; Hu, Zhenjiang

doi:10.1145/3385412.3386025

Cited by 24 publications

(13 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is similar to our input generation approach described in Section 6.6. Another approach that is closer to our work is done by Ji et al [22]. They sample the space of valid programs and encode the problem into SMT to determine an input that minimizes the number of programs that have the same output for a given input.…”

Section: Interactive Program Synthesismentioning

confidence: 99%

See 1 more Smart Citation

UnchartIt

Ramos

Pereira²,

Lynce

et al. 2020

Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

View full text Add to dashboard Cite

Charts are commonly used for data visualization. Generating a chart usually involves performing data transformations, including data pre-processing and aggregation. These tasks can be cumbersome and time-consuming, even for experienced data scientists. Reproducing existing charts can also be a challenging task when information about data transformations is no longer available.In this paper, we tackle the problem of recovering data transformations from existing charts. Given an input table and a chart, our goal is to automatically recover the data transformation program underlying the chart. We divide our approach into four steps: (1) data extraction, (2) candidate generation, (3) candidate ranking, and (4) candidate disambiguation. We implemented our approach in a tool called UnchartIt and evaluated it on a set of 50 benchmarks from Kaggle. Experimental results show that UnchartIt successfully ranks the correct data transformation among the top-10 programs in 92% of the benchmarks. To disambiguate the top-ranking programs, we use our new interactive procedure, which successfully disambiguates 98% of the ambiguous benchmarks by asking on average fewer than 2 questions to the user.

show abstract

Section: Interactive Program Synthesismentioning

confidence: 99%

“…This approach is similar to our Options model where we also minimize the number of different outputs for the same input. Our interactive approach can be seen as a generalization of Ji et al [22] work. First, we show how to formalize the optimization problem with MaxSMT.…”

Section: Interactive Program Synthesismentioning

confidence: 99%

UnchartIt

Ramos

Pereira²,

Lynce

et al. 2020

Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

View full text Add to dashboard Cite

show abstract

“…There also exist approaches that improve the generalizability by introducing user interactions as input to synthesizers [Ji et al 2020a;Mayer et al 2015;Padhi et al 2018;Wang et al 2017]. An interactive synthesizer selects inputs according to the ability to reduce the ambiguity and queries the user for the corresponding output.…”

Section: Related Workmentioning

confidence: 99%

Generalizable synthesis through unification

Xia

Xiong

et al. 2021

Proc. ACM Program. Lang.

Self Cite

View full text Add to dashboard Cite

The generalizability of PBE solvers is the key to the empirical synthesis performance. Despite the importance of generalizability, related studies on PBE solvers are still limited. In theory, few existing solvers provide theoretical guarantees on generalizability, and in practice, there is a lack of PBE solvers with satisfactory generalizability on important domains such as conditional linear integer arithmetic (CLIA). In this paper, we adopt a concept from the computational learning theory, Occam learning, and perform a comprehensive study on the framework of synthesis through unification (STUN), a state-of-the-art framework for synthesizing programs with nested if-then-else operators. We prove that Eusolver, a state-of-the-art STUN solver, does not satisfy the condition of Occam learning, and then we design a novel STUN solver, PolyGen, of which the generalizability is theoretically guaranteed by Occam learning. We evaluate PolyGen on the domains of CLIA and demonstrate that PolyGen significantly outperforms two state-of-the-art PBE solvers on CLIA, Eusolver and Euphony, on both generalizability and efficiency.

show abstract

“…Some interactive synthesis systems have question selection mechanisms to find distinguishing input [37,60]. Ji et al further approximate optimal questions to resolve ambiguity in less number of samples [31]. However, these approaches do not give generalization guarantees without assuming the existence of the target programs in the hypothesis space or a prior distribution over target programs, so they are orthogonal to our approach which works under minimal assumptions.…”

Section: Related Workmentioning

confidence: 99%

“…In program repair as well as in inductive synthesis, for example, inferring additional specifications from observed examples that must be satisfied by the program is shown to help with generalization [6,24,30,33]. Allowing the synthesizer to use more powerful oracles that adaptively craft examples or logical invariants help to synthesize correct programs [22,31]. In neural-guided program synthesis [13,18,45,54], machine learning techniques to avoid over-fitting such as regularization or structural risk minimization, are employed implicitly [59].…”

Section: Introductionmentioning

confidence: 99%

SynGuar: guaranteeing generalization in programming by example

Wang

Baluta

Kolluri

et al. 2021

Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

View full text Add to dashboard Cite

Programming by Example (PBE) is a program synthesis paradigm in which the synthesizer creates a program that matches a set of given examples. In many applications of such synthesis (e.g., program repair or reverse engineering), we are to reconstruct a program that is close to a specific target program, not merely to produce some program that satisfies the seen examples. In such settings, we wish that the synthesized program generalizes well, i.e., has as few errors as possible on the unobserved examples capturing the target function behavior. In this paper, we propose the first framework (called SynGuar) for PBE synthesizers that guarantees to achieve low generalization error with high probability. Our main contribution is a procedure to dynamically calculate how many additional examples suffice to theoretically guarantee generalization. We show how our techniques can be used in 2 well-known synthesis approaches: PROSE and STUN (synthesis through unification), for common string-manipulation program benchmarks. We find that often a few hundred examples suffice to provably bound generalization error below 5% with high (≥ 98%) probability on these benchmarks. Further, we confirm this empirically: SynGuar significantly improves the accuracy of existing synthesizers in generating the right target programs. But with fewer examples chosen arbitrarily, the same baseline synthesizers (without SynGuar) overfit and lose accuracy. CCS CONCEPTS• Software and its engineering → Software notations and tools; General programming languages.

show abstract

Question selection for interactive program synthesis

Cited by 24 publications

References 31 publications

UnchartIt

UnchartIt

Generalizable synthesis through unification

SynGuar: guaranteeing generalization in programming by example

Contact Info

Product

Resources

About