Semantic programming by example with pre-trained models

Abstract

The ability to learn programs from few examples is a powerful technology with disruptive applications in many domains, as it allows users to automate repetitive tasks in an intuitive way. Existing frameworks on inductive synthesis only perform syntactic manipulations, where they rely on the syntactic structure of the given examples and not their meaning. Any semantic manipulations, such as transforming dates, have to be manually encoded by the designer of the inductive programming framework. Recent advances in large language models have shown these models to be very adept at performing semantic transformations of its input by simply providing a few examples of the task at hand. When it comes to syntactic transformations, however, these models are limited in their expressive power. In this paper, we propose a novel framework for integrating inductive synthesis with few-shot learning language models to combine the strength of these two popular technologies. In particular, the inductive synthesis is tasked with breaking down the problem in smaller subproblems, among which those that cannot be solved syntactically are passed to the language model. We formalize three semantic operators that can be integrated with inductive synthesizers. To minimize invoking expensive semantic operators during learning, we introduce a novel deferred query execution algorithm that considers the operators to be oracles during learning. We evaluate our approach in the domain of string transformations: the combination methodology can automate tasks that cannot be handled using either technologies by themselves. Finally, we demonstrate the generality of our approach via a case study in the domain of string profiling.

ScatterShot: Interactive In-context Example Curation for Text Transformation

“…Temporal expression extraction and normalization. The Temporal task involves data wrangling [60], where the goal is extracting phrases with temporal expressions from sentences or documents, and normalizing them into a standard format [9]. As shown in Figure 1, these can include absolute or relative dates, and can have different granularity (e.g., exact date vs. year only).…”

WikiCoder: Learning to Write Knowledge-Powered Code

“…The second example, inspired from [22], is about language learning: tasks are grammar exercises, where the goal is to write a grammatically correct sentence. Here the knowledge graph includes grammatical forms and their connections, such as verbs and their different conjugated forms, pronouns, adjectives, and so on.…”

“…Knowledge-powered program synthesis extends classical program synthesis by targetting knowledge-powered programs. The challenge of combining syntactical manipulations performed in program synthesis with semantical information was recently set out by [22]. They discuss a number of applications: string manipulations, code refactoring, and string profiling, and construct an algorithm based on very large language models (see related work section).…”

Example-based Synthesis of Static Analysis Rules

“…In the domain of synthesizing String transformations, pioneered by FlashFill [Gulwani 2011], RobustFill [Devlin et al 2017a] presents a neural approach for program synthesis as well as program induction. Hybrid approaches such as [Gulwani and Jain 2017;Kalyan et al 2018;Verbruggen et al 2021] complement machine learning based induction with algorithmic techniques and present an interesting direction for exploration in the context of rule synthesis.…”