Collecting and Characterizing Natural Language Utterances for Specifying Data Visualizations

Abstract

Natural language interfaces (NLIs) for data visualization are becoming increasingly popular both in academic research and in commercial software. Yet, there is a lack of empirical understanding of how people specify visualizations through natural language.We conducted an online study (N = 102), showing participants a series of visualizations and asking them to provide utterances they would pose to generate the displayed charts. From the responses, we curated a dataset of 893 utterances and characterized the utterances according to (1) their phrasing (e.g., commands, queries, questions) and ( 2) the information they contained (e.g., chart types, data aggregations). To help guide future research and development, we contribute this utterance dataset and discuss its applications toward the creation and benchmarking of NLIs for visualization. CCS CONCEPTS• Human-centered computing → Visualization; Empirical studies in visualization; Natural language interfaces.

How do you Converse with an Analytical Chatbot? Revisiting Gricean Maxims for Designing Analytical Conversational Behavior

“…Given that the application domain for these chatbot interactions uses a set of known analytical intents along with attributes and values from the underlying data, the space of linguistic variations is relatively small and the outputs can be specified using templates [55]. We define the templates by referring to utterances from Study 1, along with utterances commonly supported across existing NLIs [40,52,62,64,76] and sample utterances collected through studies investigating the use of NL to create or interact with data visualizations [68,72]. The grammar rules from the parser modules are used to aid in the NLG process, which involves ordering constituents of the NLG output and The Slack chatbot uses the Slack API [11] for listening to Slack events.…”

Synchromesh: Reliable code generation from pre-trained language models

“…For SQL, we use the Spider dataset (Yu et al, 2018). For Vega-Lite, we use the NLV Corpus (Srinivasan et al, 2021). For SMCalFlow, we use the dataset that introduced the language (Andreas et al, 2020).…”

Snowy: Recommending Utterances for Conversational Visual Analysis

“…Having a deterministic set of generated NL output also allowed us to control the variability in the recommended NL utterances for testing purposes. We defined the templates by referring to utterances commonly supported across existing NLIs [35,46,54,57,79] through studies investigating the use of NL to create or interact with data visualizations [62,67]. Note, however, that the current template-based approach can be extended to a task-oriented dialogue approach by using the set of templates along with a language model for generating a larger variety of sentences with linguistic variability.…”