Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code

Pan, Rangeet; Ibrahimzada, Ali Reza; Krishna, Rahul; Sankar, Divya; Wassi, Lambert Pouguem; Merler, Michele; Sobolev, Boris; Pavuluri, Raju; Sinha, Saurabh; Jabbarvand, Reyhaneh

doi:10.1145/3597503.3639226

Cited by 14 publications

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Type-migrating C-to-Rust translation using a large language model

Hong,

Ryu

2024

Empir Software Eng

View full text Add to dashboard Cite

Rust, a modern system programming language, introduces new types that prevent memory bugs and data races. This makes translating legacy system programs from C to Rust a promising approach to enhance their reliability. Since manual code translation is time-consuming, it is desirable to automate the translation. To yield satisfactory results, the translator should have the ability to perform type migration, i.e., removing C types and introducing Rust types in the code. In this work, we aim to automatically port an entire C program to Rust by translating each C function to a Rust function with a signature containing proper Rust types through type migration. This goal is challenging because (1) type migration cannot be achieved through syntactic mappings between type names, and (2) after type migration, function bodies should be correctly restructured based on the precise understanding of the functions’ behavior. To address these difficulties, we leverage large language models (LLMs), which possess knowledge of program semantics and programming idioms. However, naïvely instructing LLMs to translate each function produces unsatisfactory Rust code, containing unmigrated or improperly migrated types and a huge number of type errors. To resolve these issues, we propose three techniques: (1) generating candidate signatures, (2) providing translated callees’ signatures to LLMs, and (3) iteratively fixing type errors using compiler feedback. Our evaluation shows that the proposed approach yields a 63.5% increase in migrated types and a 71.5% decrease in type errors compared to the baseline (the naïve LLM-based translation) with modest performance overhead.

show abstract