Better Together? An Evaluation of AI-Supported Code Translation

Abstract

Generative machine learning models have recently been applied to source code, for use cases including translating code between programming languages, creating documentation from code, and auto-completing methods. Yet, state-of-the-art models often produce code that is erroneous or incomplete. In a controlled study with 32 software engineers, we examined whether such imperfect outputs are helpful in the context of Java-to-Python code translation. When aided by the outputs of a code translation model, participants produced code with fewer errors than when working alone. We also examined how the quality and quantity of AI translations affected the work process and quality of outcomes, and observed that providing multiple translations had a larger impact on the translation process than varying the quality of provided translations. Our results tell a complex, nuanced story about the benefits of generative code models and the challenges software engineers face when working with their outputs. Our work motivates the need for intelligent user interfaces that help software engineers effectively work with generative code models in order to understand and evaluate their outputs and achieve superior outcomes to working alone.CCS Concepts: • Human-centered computing → HCI theory, concepts and models; • Software and its engineering → Designing software; • Computing methodologies → Generative and developmental approaches.

The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development

“…Empirical evaluations of this model have shown that, although the quality of its outputs is quite good, those outputs may still be problematic [57]. Echoing the results from Weisz et al [103], human-centered evaluations of Copilot have found that it increases users' feelings of productivity [109], and that almost a third (27%) of its proposed code completions were accepted by users. In a contrasting evaluation, Vaithilingam et al [95] found that while most participants expressed a preference to use Copilot in their daily work, it did not necessarily improve their task completion times or success rates.…”

“…code that is free of syntax or logical errors). Nonetheless, Weisz et al [102] found that software engineers are still interested in using such models in their work, and that the imperfect outputs of these models can even help them produce higher-quality code via human-AI collaboration [103].…”

“…We note that prior studies (e.g. [103,105,109]) conducted quantitative examinations of the use of LLMs in code work; our study is akin to Weisz et al's qualitative examination of software engineers' attitudes toward working with models that may fail to produce working code [102].…”

“…Our scale items were modeled from scales published in Weisz et al[103, Table 9 -AI Support] that measured constructs including ease of use (item 3), response quality (item 1), the production of higher-quality code (item 5), and the ability to write code more rapidly (item 4). We added additional items to cover the constructs of request understanding and enjoyment, and we cast all items on a 4-point scale of extent.…”

Intelligent User Interfaces and Their Evaluation: A Systematic Mapping Study

“…Factors observed in studies evaluating UX [ 21 ] include appearance, perceptions, performance, availability, and overall satisfaction. Additionally, cognitive load [ 22 ] and efficacy [ 23 ] have also been observed as factors in intelligent user interface evaluation.…”