What is it like to program with artificial intelligence?

Sarkar, Advait; Gordon, Andrew; Negreanu, Carina; Poelitz, Christian; Ragavan, Sruti Srinivasa; Zorn, Ben

doi:10.48550/arxiv.2208.06213

Cited by 13 publications

(13 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sarkar et al [95] review studies of the usability of natural language programming with LLMs in particular (e.g., [43,105,112,123,124]) and articulate how LLMs have made the abstraction matching problem "fuzzy" (termed "fuzzy abstraction matching"): while LLMs can interpret a much wider variety of naturalistic utterances than earlier models, as a consequence, the space of utterances that may be effective at controlling the model is even more difficult and unpredictable for an end-user to learn. While a previous model might simply fail to interpret an utterance, an LLM may interpret it in a manner that is opaque and difficult for the user to generalize from.…”

Section: Natural Language Programmingmentioning

confidence: 99%

“…We conducted an informal, exploratory study to gain an initial understanding of the challenges users might face when interacting with an LLM for data analysis in spreadsheets, and to check whether and to what extent the abstraction matching challenges observed by Sarkar et al [95] and Srinivasa Ragavan et al [105] were present in our application.…”

Section: Formative Usability Studymentioning

confidence: 99%

“…Consider the variety of ways in which one can express even the relatively simple intent to set a 2-minute timer and the small number of utterances that typically produce the correct result in contemporary voice assistants. Choosing the right level of abstraction, the right level of granularity in a command, is called the abstraction matching problem [95].…”

Section: Natural Language Interfacesmentioning

confidence: 99%

“…Despite impressive benchmark performance, LLMs are beset with issues in practical use. Lab and field studies have shown that the mapping between natural language and code is poorly understood, that generated code can contain subtle bugs, and that generated code can be difficult to verify [95,112,124].…”

Section: Introductionmentioning

confidence: 99%

“…In this paper, we consider the specific problem of abstraction matching [95]: when the user has a well-formed intent, how do they select an utterance from the near infinite space of naturalistic utterances that they believe the system will reliably map to a satisfactory solution? This involves "matching" the utterance to the right level of "abstraction", by specifying the utterance at a level of granularity and detail that matches the set of actions the system can take, and selecting suitable words and grammar.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

“What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models

Liu¹,

Sarkar²,

Negreanu³

et al. 2023

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Code-generating large language models map natural language to code. However, only a small portion of the infinite space of naturalistic utterances is effective at guiding code generation. For non-expert end-user programmers, learning this is the challenge of abstraction matching. We examine this challenge in the specific context of data analysis in spreadsheets, in a system that maps the user's natural language query to Python code using the Codex generator, executes the code, and shows the result. We propose grounded abstraction matching, which bridges the abstraction gap by translating the code back into a systematic and predictable naturalistic utterance. In a between-subjects, think-aloud study (n=24), we compare grounded abstraction matching to an ungrounded alternative based on previously established query framing principles. We find that the grounded approach improves end-users' understanding of the scope and capabilities of the code-generating model, and the kind of language needed to use it effectively. CCS CONCEPTS• Human-centered computing → Natural language interfaces; Interactive systems and tools; Empirical studies in HCI .

show abstract