Survey of Compiler Testing Methods

Kossatchev, A. S.; Posypkin, Mikhail

doi:10.1007/s11086-005-0002-z

Cited by 12 publications

(10 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…29 Automated Compiler Testing. The idea of randomly generating or mutating programs to induce errors in production compilers and interpreters has a long history, with grammar-or mutationbased fuzzers having been designed to test implementations of languages such as COBOL [Sauder 1962], PL/I [Hanford 1970], FORTRAN [Burgess and Saidi 1996], Ada and Pascal [Wichmann 1998], and more recently C [Le et al 2014[Le et al , 2015aNagai et al 2014;Nakamura and Ishiura 2016;Sun et al 2016a;Yang et al 2011;Yarpgen 2018], JavaScript and PHP [Holler et al 2012], Java byte-code [Chen et al 2016], OpenCL [Lidbury et al 2015], GLSL [Donaldson et al 2017;Donaldson and Lascu 2016] and C++ [Sun et al 2016b] (see also two surveys on the topic [Boujarwah and Saleh 1997;Kossatchev and Posypkin 2005]). Related approaches have been used to test other programming language processors, such as static analysers , refactoring engines [Daniel et al 2007], and symbolic executors [Kapus and Cadar 2017].…”

Section: Related Workmentioning

confidence: 99%

“…test generation has been a topic of interest for many years (see e.g. [Boujarwah and Saleh 1997;Hanford 1970; Kossatchev and Posypkin 2005;Purdom 1972;Sauder 1962;Wichmann 1998]), and recent years have seen the development of several compiler fuzzing tools that employ randomised testing to search for bugs in (primarily C) compilers [Le et al 2014;Nakamura and Ishiura 2016;Yang et al 2011;Yarpgen 2018]. Such compiler fuzzers have practically demonstrated their ability to find hundreds of bugs, and particularly many miscompilations, in widely-used compilers.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Compiler fuzzing: how much does it matter?

Marcozzi

Tang

Donaldson

et al. 2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

Despite much recent interest in randomised testing (fuzzing) of compilers, the practical impact of fuzzer-found compiler bugs on real-world applications has barely been assessed. We present the first quantitative and qualitative study of the tangible impact of miscompilation bugs in a mature compiler. We follow a rigorous methodology where the bug impact over the compiled application is evaluated based on (1) whether the bug appears to trigger during compilation; (2) the extent to which generated assembly code changes syntactically due to triggering of the bug; and (3) whether such changes cause regression test suite failures, or whether we can manually find application inputs that trigger execution divergence due to such changes. The study is conducted with respect to the compilation of more than 10 million lines of C/C++ code from 309 Debian packages, using 12% of the historical and now fixed miscompilation bugs found by four state-of-the-art fuzzers in the Clang/LLVM compiler, as well as 18 bugs found by human users compiling real code or as a by-product of formal verification efforts. The results show that almost half of the fuzzer-found bugs propagate to the generated binaries for at least one package, in which case only a very small part of the binary is typically affected, yet causing two failures when running the test suites of all the impacted packages. User-reported and formal verification bugs do not exhibit a higher impact, with a lower rate of triggered bugs and one test failure. The manual analysis of a selection of the syntactic changes caused by some of our bugs (fuzzer-found and non fuzzer-found) in package assembly code, shows that either these changes have no semantic impact or that they would require very specific runtime circumstances to trigger execution divergence. CCS Concepts: • Software and its engineering → Compilers; Software verification and validation.

show abstract

Section: Related Workmentioning

confidence: 99%

mentioning

confidence: 99%

Compiler fuzzing: how much does it matter?

Marcozzi

Tang

Donaldson

et al. 2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

show abstract

“…Compiler testing is a difficult and essential task due to the complexity and importance of compilers, and thus it has attracted extensive attentions from researchers [37], [38], [39], [40], [41]. Research on compiler testing can be divided into three main aspects, i.e., test-program generation, testoracle construction, and test-execution optimization.…”

Section: Related Work 81 Compiler Testingmentioning

confidence: 99%

Coverage Prediction for Accelerating Compiler Testing

Chen

Wang

Hao

et al. 2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Compilers are one of the most fundamental software systems. Compiler testing is important for assuring the quality of compilers. Due to the crucial role of compilers, they have to be well tested. Therefore, automated compiler testing techniques (those based on randomly generated programs) tend to run a large number of test programs (which are test inputs of compilers). The cost for compilation and execution for these test programs is significant. These techniques can take a long period of testing time to detect a relatively small number of compiler bugs. That may cause many practical problems, e.g., bringing a lot of costs including time costs and financial costs, and delaying the development/release cycle. Recently, some approaches have been proposed to accelerate compiler testing by executing test programs that are more likely to trigger compiler bugs earlier according to some criteria. However, these approaches ignore an important aspect in compiler testing: different test programs may have similar test capabilities (i.e., testing similar functionalities of a compiler, even detecting the same compiler bug), which may largely discount their acceleration effectiveness if the test programs with similar test capabilities are executed all the time. Test coverage is a proper approximation to help distinguish them, but collecting coverage dynamically is infeasible in compiler testing since most test programs are generated on the fly by automatic test-generation tools like Csmith. In this paper, we propose the first method to predict test coverage statically for compilers, and then propose to prioritize test programs by clustering them according to the predicted coverage information. The novel approach to accelerating compiler testing through coverage prediction is called COP (short for COverage Prediction). Our evaluation on GCC and LLVM demonstrates that COP significantly accelerates compiler testing, achieving an average of 51.01% speedup in test execution time on an existing dataset including three old release versions of the compilers and achieving an average of 68.74% speedup on a new dataset including 12 latest release versions. Moreover, COP outperforms the state-of-the-art acceleration approach significantly by improving 17.16%∼82.51% speedups in different settings on average.

show abstract

“…Another related problem is testing and comparing code compilers , although in that case, research focused on generating test programs from syntactic and semantic definition rules.…”

Section: Related Workmentioning

confidence: 99%

Evaluating code‐based test input generator tools

Cseppento

Micskei

2017

Software Testing Verif & Rel

View full text Add to dashboard Cite

Summary In recent years, several tools have been developed to automatically select test inputs from the code of the system under test. However, each of these tools has different advantages, and there is a little detailed feedback available on the actual capabilities of the various tools. To evaluate test input generators, this paper collects a set of programming language concepts that should be handled by the tools and maps these core concepts and challenging features like handling the environment or multi‐threading to 363 code snippets, respectively. These snippets would serve as inputs for the tools. Next, the paper presents SETTE, an automated framework to execute and evaluate these snippets. Using SETTE, multiple experiments were performed on five Java and one .NET‐based tools using symbolic execution, search‐based, and random techniques. The test suites' coverage, size, generation time, and mutation score were compared. The results highlight the strengths and weaknesses of each tool and approach and identify hard code parts that are difficult to tackle for most of the tools. We hope that this research could serve as actionable feedback to tool developers and help practitioners assess the readiness of test input generation.

show abstract

Survey of Compiler Testing Methods

Cited by 12 publications

References 8 publications

Compiler fuzzing: how much does it matter?

Compiler fuzzing: how much does it matter?

Coverage Prediction for Accelerating Compiler Testing

Evaluating code‐based test input generator tools

Contact Info

Product

Resources

About