PlotCoder: Hierarchical Decoding for Synthesizing Visualization Code in Programmatic Context

Abstract

Creating effective visualization is an important part of data analytics. While there are many libraries for creating visualizations, writing such code remains difficult given the myriad of parameters that users need to provide. In this paper, we propose the new task of synthesizing visualization programs from a combination of natural language utterances and code context. To tackle the learning problem, we introduce PLOTCODER, a new hierarchical encoder-decoder architecture that models both the code context and the input utterance. We use PLOTCODER to first determine the template of the visualization code, followed by predicting the data to be plotted. We use Jupyter notebooks containing visualization programs crawled from GitHub to train PLOTCODER. On a comprehensive set of test samples from those notebooks, we show that PLOTCODER correctly predicts the plot type of about 70% samples, and synthesizes the correct programs for 35% samples, performing 3-4.5% better than the baselines. 1

School of Mechanical Engineering, Shanghai Jiao Tong University, 200240, Shanghai, China

“…Euphony (Lee et al, 2018), Neo (Feng et al, 2018) LPN (Ling et al, 2016), NSPS (Parisotto et al, 2017), DeepCoder (Balog et al, 2017), Robust-Fill (Devlin et al, 2017a), (Yin & Neubig, 2017), ASN (Rabinovich et al, 2017), NGDS (Kalyan et al, 2018), Bunel et al (2018), ReCode (Hayati et al, 2018), Au-toPandas (Bavishi et al, 2019), , PlotCoder (Chen et al, 2021c), TreeGen , RED-CODER (Parvez et al, 2021), Jigsaw (Jain et al, 2022), JuPyT5 (Chandel et al, 2022), CodeT (Chen et al, 2023a), TiCoder (Lahiri et al, 2022, AceCoder (Li et al, 2023e), Self-Debugging (Chen et al, 2023c), Clari-fyGPT (Mu et al, 2023) Text-to-SQL Seq2SQL , SQLNet (Xu et al, 2017), Suhr et al (2018), Type-SQL (Yu et al, 2018a), Coarse2Fine (Dong & Lapata, 2018), , SyntaxSQL-Net (Yu et al, 2018b), GNN (Bogin et al, 2019), TREQS (Wang et al, 2020b) SQLova (Hwang et al, 2019), IRNet , , RAT-SQL (Wang et al, 2020a), Bertrand-DR (Kelkar et al, 2020), RYANSQL (Choi et al, 2021), TaBERT (Yin et al, 2020), Photon (Zeng et al, 2020), HydraNet , GAZP , GraPPa ), SmBoP (Rubin & Berant, 2021, NQG-T5…”

A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspective

“…Zhong et al (2017) develop an LSTM-based sequence to sequence model adapted from Dong and Lapta (2016) with a pointer network (Vinyals et al, 2015) containing augmented inputs specific to the SQL language. Chen et al (2021b) propose PlotCoder, an LSTM-based Seq2Seq with a copying mechanism trained on a portion on the JuICe dataset (Agashe et al, 2019) trained to generate plot code snippets in Python Liguori et al (2021a) use a standard LSTM-based sequence to sequence model to generate assembly snippets from natural language intents that are then compiled into shellcodes used in exploiting vulnerabilities. Frempong et al (2021) generate JavaScript XSS exploits from natural language intents.…”

Active Programming by Example with a Natural Language Prior

“…• the SQL program s can be generalized to other types of executable semantic parses, such as tensor manipulation commands, visualization programs (Chen et al, 2021c), or dataflow graphs (Semantic Machines et al, 2020);…”