Active Programming by Example with a Natural Language Prior

Abstract

We introduce APEL, a new framework that enables non-programmers to indirectly annotate natural language utterances with executable meaning representations, such as SQL programs. Based on a natural language utterance, we first run a seed semantic parser to generate a prior over a list of candidate programs. To obtain information about which candidate is correct, we synthesize an input on which the more likely programs tend to produce different outputs, and ask an annotator which output is appropriate for the utterance. Hence, the annotator does not have to directly inspect the programs. To further reduce effort required from annotators, we aim to synthesize simple input databases that nonetheless have high information gain. With human annotators and Bayesian inference to handle annotation errors, we outperform Codex's top-1 performance (59%) and achieve the same accuracy as the original expert annotators (75%), by soliciting answers for each utterance on only 2 databases with an average of 9 records each. In contrast, it would be impractical to solicit outputs on the original 30K-record databases provided by SPIDER.

Evaluating Cross-Domain Text-to-SQL Models and Benchmarks

“…Generally, a query that is equivalent (but not identical) to ground truth may be mistakenly classified as incorrect by automated evaluation metrics. Another study by Zhong et al (2022) identifies limitations within the Spider benchmark, such as issues with ties and certain syntactic problems. Their analysis is primarily focused on a subset of Spider, without quantifying the extent or impact of these limitations or conducting an assessment of other benchmarks.…”

InstructExcel: A Benchmark for Natural Language Instruction in Excel

“…Other important aspects studied include length generalization (Anil et al, 2022), compositional generalization (Shi et al, 2022), reverse engineering (Pearce et al, 2022), and generating development tools (Bareiß et al, 2022). The task of NL to Code is broadly of interest to the semantic parsing literature (Kamath and Das, 2018;Zhong et al, 2022;.…”