Toward understanding compiler bugs in GCC and LLVM

Sun, Chengnian; Le, Vu; Zhang, Qirun; Su, Zhendong

doi:10.1145/2931037.2931074

Cited by 92 publications

(42 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Understanding Compiler Bugs. A recent empirical study provides an in-depth analysis of bugs in the GCC and LLVM compilers [Sun et al 2016b], focusing on aspects such as the distribution of bugs across compiler components, the sizes of triggering test cases associated with bug reports, the lifetime of bug reports from filing to closing, and the developer-assigned priority levels for bugs and how these correlate to compiler components. The study is complementary to ours: beyond a discussion of bug priorities, it is not concerned with the extent to which compiler bugs affect realworld applications, and it does not focus on whether the bugs under analysis are miscompilations, nor whether the bugs were found in the wild or via automated tools such as fuzzers.…”

Section: Related Workmentioning

confidence: 99%

“…29 Automated Compiler Testing. The idea of randomly generating or mutating programs to induce errors in production compilers and interpreters has a long history, with grammar-or mutationbased fuzzers having been designed to test implementations of languages such as COBOL [Sauder 1962], PL/I [Hanford 1970], FORTRAN [Burgess and Saidi 1996], Ada and Pascal [Wichmann 1998], and more recently C [Le et al 2014[Le et al , 2015aNagai et al 2014;Nakamura and Ishiura 2016;Sun et al 2016a;Yang et al 2011;Yarpgen 2018], JavaScript and PHP [Holler et al 2012], Java byte-code [Chen et al 2016], OpenCL [Lidbury et al 2015], GLSL [Donaldson et al 2017;Donaldson and Lascu 2016] and C++ [Sun et al 2016b] (see also two surveys on the topic [Boujarwah and Saleh 1997;Kossatchev and Posypkin 2005]). Related approaches have been used to test other programming language processors, such as static analysers , refactoring engines [Daniel et al 2007], and symbolic executors [Kapus and Cadar 2017].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Compiler fuzzing: how much does it matter?

Marcozzi

Tang

Donaldson

et al. 2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

Despite much recent interest in randomised testing (fuzzing) of compilers, the practical impact of fuzzer-found compiler bugs on real-world applications has barely been assessed. We present the first quantitative and qualitative study of the tangible impact of miscompilation bugs in a mature compiler. We follow a rigorous methodology where the bug impact over the compiled application is evaluated based on (1) whether the bug appears to trigger during compilation; (2) the extent to which generated assembly code changes syntactically due to triggering of the bug; and (3) whether such changes cause regression test suite failures, or whether we can manually find application inputs that trigger execution divergence due to such changes. The study is conducted with respect to the compilation of more than 10 million lines of C/C++ code from 309 Debian packages, using 12% of the historical and now fixed miscompilation bugs found by four state-of-the-art fuzzers in the Clang/LLVM compiler, as well as 18 bugs found by human users compiling real code or as a by-product of formal verification efforts. The results show that almost half of the fuzzer-found bugs propagate to the generated binaries for at least one package, in which case only a very small part of the binary is typically affected, yet causing two failures when running the test suites of all the impacted packages. User-reported and formal verification bugs do not exhibit a higher impact, with a lower rate of triggered bugs and one test failure. The manual analysis of a selection of the syntactic changes caused by some of our bugs (fuzzer-found and non fuzzer-found) in package assembly code, shows that either these changes have no semantic impact or that they would require very specific runtime circumstances to trigger execution divergence. CCS Concepts: • Software and its engineering → Compilers; Software verification and validation.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Compiler fuzzing: how much does it matter?

Marcozzi

Tang

Donaldson

et al. 2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

show abstract

“…For example, for GCC in Table 10, four of ten are tree-optimization-related files, and we can find that a large number of GCC bugs occurring at tree optimization parts from GCC Bugzilla 11 . Similarly, for LLVM in Table 11, "X86ISelLowering.cpp" and "SemaExprCXX.cpp" are reported as the two of top-10 files containing the largest number of bugs [34]. The highly accurate prediction for such files facilitates to separate test programs triggering more different bugs into different groups, so that COP accelerates compiler testing to a larger extent.…”

Section: Why Cop Workmentioning

confidence: 99%

Coverage Prediction for Accelerating Compiler Testing

Chen

Wang

Hao

et al. 2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Compilers are one of the most fundamental software systems. Compiler testing is important for assuring the quality of compilers. Due to the crucial role of compilers, they have to be well tested. Therefore, automated compiler testing techniques (those based on randomly generated programs) tend to run a large number of test programs (which are test inputs of compilers). The cost for compilation and execution for these test programs is significant. These techniques can take a long period of testing time to detect a relatively small number of compiler bugs. That may cause many practical problems, e.g., bringing a lot of costs including time costs and financial costs, and delaying the development/release cycle. Recently, some approaches have been proposed to accelerate compiler testing by executing test programs that are more likely to trigger compiler bugs earlier according to some criteria. However, these approaches ignore an important aspect in compiler testing: different test programs may have similar test capabilities (i.e., testing similar functionalities of a compiler, even detecting the same compiler bug), which may largely discount their acceleration effectiveness if the test programs with similar test capabilities are executed all the time. Test coverage is a proper approximation to help distinguish them, but collecting coverage dynamically is infeasible in compiler testing since most test programs are generated on the fly by automatic test-generation tools like Csmith. In this paper, we propose the first method to predict test coverage statically for compilers, and then propose to prioritize test programs by clustering them according to the predicted coverage information. The novel approach to accelerating compiler testing through coverage prediction is called COP (short for COverage Prediction). Our evaluation on GCC and LLVM demonstrates that COP significantly accelerates compiler testing, achieving an average of 51.01% speedup in test execution time on an existing dataset including three old release versions of the compilers and achieving an average of 68.74% speedup on a new dataset including 12 latest release versions. Moreover, COP outperforms the state-of-the-art acceleration approach significantly by improving 17.16%∼82.51% speedups in different settings on average.

show abstract

“…Without anyways for solving this problemdevelopers have to make «The assumption of the correctness of the compiler». According to a study [4] conducted in 2016, the total number of bugs found in GCC+LLVM are more than 50000. This is one of the main reasons why this assumption may not be sufficient for critical systems.…”

Section: Introductionmentioning

confidence: 99%

Applying High-Level Function Loop Invariants for Machine Code Deductive Verification

Путро¹

2019

Proceedings of ISP RAS

View full text Add to dashboard Cite

The existing tools of deductive verification allow us to successfully prove the correctness of functions written in high-level languages such as C or Java. However, this may not be enough for critical software because even fully verified code cannot guarantee the correct generation of machine code by the compiler. At the moment, developers of such systems have to accept the compiler correctness assumption, which, however, is extremely undesirable, but inevitable due to the lack of full-fledged systems of formal verification of machine code. It is also worth noting that the verification of machine code by a person directly is an extremely time-consuming task due to the high complexity and large amounts of machine code. One of the approaches to simplify the verification of machine code is automatic deductive verification reusing the formal specification of the high-level language function. The formal specification of the function consists of the specification of the pre-and postcondition, as well as loop invariants, which specify conditions that are satified at each iteration of the loop. When compiling a program into machine code, pre-and postconditions are preserved, which, however, cannot be said about loop invariants. This fact is one of the main problems of automatic verification of machine code with loops. Another important problem is that high-level function variables often have 'projections' to both registers and memory at the machine code level, and the verification procedure requires that invariants be described for each variable, and therefore the missing invariants must be generated. This paper presents an approach to solving these problems, based on the analysis of the control flow graph, and intelligent search of the locations of variables.

show abstract

Toward understanding compiler bugs in GCC and LLVM

Cited by 92 publications

References 29 publications

Compiler fuzzing: how much does it matter?

Compiler fuzzing: how much does it matter?

Coverage Prediction for Accelerating Compiler Testing

Applying High-Level Function Loop Invariants for Machine Code Deductive Verification

Contact Info

Product

Resources

About