Synchromesh: Reliable code generation from pre-trained language models

Poesia, Gabriel; Polozov, Oleksandr; Le, Vu; Tiwari, Ashish; Soares, Gustavo; Meek, Christopher; Gulwani, Sumit

doi:10.48550/arxiv.2201.11227

Cited by 15 publications

(24 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast to most existing neuro-symbolic reasoning frameworks, e.g., [24], instead of using a pretrained or jointly-trained semantic parser, we introduce the use of large language-to-code models for parsing. Specifically, we use Codex [6,11] with the Synchromesh framework [27]. By specifying only a small number of examples of language input and expected programs, we gain perfect parsing capabilities across unseen categories and re-…”

Section: Semantic Parsermentioning

confidence: 99%

NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations

Hsu¹,

Mao²

2023

Preprint

View full text Add to dashboard Cite

Grounding object properties and relations in 3D scenes is a prerequisite for a wide range of artificial intelligence tasks, such as visually grounded dialogues and embodied manipulation. However, the variability of the 3D domain induces two fundamental challenges: 1) the expense of labeling and 2) the complexity of 3D grounded language. Hence, essential desiderata for models are to be data-efficient, generalize to different data distributions and tasks with unseen semantic forms, as well as ground complex language semantics (e.g., view-point anchoring and multi-object reference). To address these challenges, we propose NS3D, a neuro-symbolic framework for 3D grounding. NS3D translates language into programs with hierarchical structures by leveraging large language-to-code models. Different functional modules in the programs are implemented as neural networks. Notably, NS3D extends prior neuro-symbolic visual reasoning methods by introducing functional modules that effectively reason about high-arity relations (i.e., relations among more than two objects), key in disambiguating objects in complex 3D scenes. Modular and compositional architecture enables NS3D to achieve state-of-the-art results on the ReferIt3D view-dependence task, a 3D referring expression comprehension benchmark. Importantly, NS3D shows significantly improved performance on settings of data-efficiency and generalization, and demonstrate zero-shot transfer to an unseen 3D question-answering task.

show abstract

Section: Semantic Parsermentioning

confidence: 99%

NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations

Hsu¹,

Mao²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Usually this is accomplished by semantic retrieval using the testing input as the query. Previous works have studied how to build sentence-level retrievers (Poesia et al, 2022;Rubin et al, 2021). Our work goes beyond sentences and studies how to retrieve dialogues.…”

Section: Dialogue Retrievermentioning

confidence: 99%

“…The first is similarity-based retrieval. Poesia et al (2022) and Das et al (2021) define a similarity metric between semantic parsing results and use this similarity as the training objective for the retriever.…”

Section: Related Workmentioning

confidence: 99%

In-Context Learning for Few-Shot Dialogue State Tracking

Hu¹,

Lee²,

Xie³

et al. 2022

Preprint

View full text Add to dashboard Cite

Collecting and annotating task-oriented dialogues is time-consuming and costly. Thus, few-shot learning for dialogue tasks presents an exciting opportunity. In this work, we propose an in-context (IC) learning framework for few-shot dialogue state tracking (DST), where a large pre-trained language model (LM) takes a test instance and a few annotated examples as input, and directly decodes the dialogue states without any parameter updates. This makes the LM more flexible and scalable compared to prior few-shot DST work when adapting to new domains and scenarios. We study ways to formulate dialogue context into prompts for LMs, and propose an efficient approach to retrieve dialogues as exemplars given a test instance and a selection pool of few-shot examples. To better leverage the pre-trained LMs, we also reformulate DST into a text-to-SQL problem. Empirical results on MultiWOZ 2.1 and 2.4 show that our method IC-DST outperforms previous fine-tuned state-of-the-art models in few-shot settings. 1

show abstract

“…The study also suggested the future direction of this research domain to improve automatic code generation using natural language by analyzing the current trend of approaches. [30] proposes Synchromesh, a framework to improve the coding reliability of pre-trained models. Using Target Similarity Tuning, this framework retrieves a few-shot example from a training bank.…”

Section: Ptm For Code Generation and Understandingmentioning

confidence: 99%

On the Effectiveness of Pretrained Models for API Learning

Hadi,

Yusuf,

Thung

et al. 2022

Preprint

View full text Add to dashboard Cite

Developers frequently use APIs to implement certain functionalities, such as parsing Excel Files, reading and writing text files line by line, etc. Developers can greatly benefit from automatic API usage sequence generation based on natural language queries for building applications in a faster and cleaner manner. Existing approaches utilize information retrieval models to search for matching API sequences given a query or use RNN-based encoder-decoder to generate API sequences. As it stands, the first approach treats queries and API names as bags of words. It lacks deep comprehension of the semantics of the queries. The latter approach adapts a neural language model to encode a user query into a fixed-length context vector and generate API sequences from the context vector.We want to understand the effectiveness of recent Pre-trained Transformer based Models (PTMs) for the API learning task. These PTMs are trained on large natural language corpora in an unsupervised manner to retain contextual knowledge about the language and have found success in solving similar Natural Language Processing (NLP) problems. However, the applicability of PTMs has not yet been explored for the API sequence generation task. We use a dataset that contains 7 million annotations collected from GitHub to evaluate the PTMs empirically. This dataset was also used to assess previous approaches. Based on our results, PTMs generate more accurate API sequences and outperform other related methods by ∼11%. We have also identified two different tokenization approaches that can contribute to a significant boost in PTMs' performance for the API sequence generation task.

show abstract

Synchromesh: Reliable code generation from pre-trained language models

Cited by 15 publications

References 13 publications

NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations

NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations

In-Context Learning for Few-Shot Dialogue State Tracking

On the Effectiveness of Pretrained Models for API Learning

Contact Info

Product

Resources

About