Randomized stress-testing of link-time optimizers

Le, Vu; Sun, Chengnian; Su, Zhendong

doi:10.1145/2771783.2771785

Cited by 44 publications

(8 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this study, we use the two most widely-used C compilers, i.e., GCC and LLVM for the x86 64-Linux platform, which cover almost all C compilers used in the study of C compiler testing [1], [3], [4], [5], [16], [27]. More specifically, we use the constructed dataset (i.e., GCC-4.4.3, LLVM-2.6, and LLVM-2.7) in the existing work [6] to evaluate the effectiveness of COP on accelerating compiler testing, which is convenient to compare with the state-of-the-art acceleration approach LET.…”

Section: Subjects and Test Programsmentioning

confidence: 99%

“…Also, Le et al [17], Sun et al [18], and Zhang et al [53] further proposed more mutation strategies to generate test programs based on existing test programs. Following the existing work [1], [3], [5], [6], [17], [27], our work uses the test programs generated by Csmith to test compilers. Please note that our approach COP aims to accelerate compiler testing by prioritizing the original test programs generated by Csmith rather than generate any new test programs.…”

Section: Related Work 81 Compiler Testingmentioning

confidence: 99%

“…In particular, differential testing needs two or more comparable compilers and determines whether some compilers have bugs by comparing the results produced by these compilers. Based on differential testing, Le et al [27] proposed to detect the bugs of link-time optimizers in compilers by randomized stress-testing. Besides, Le et al [4] proposed equivalence modulo inputs (EMI) to address the test-oracle construction problem.…”

Section: Related Work 81 Compiler Testingmentioning

confidence: 99%

See 2 more Smart Citations

Coverage Prediction for Accelerating Compiler Testing

Chen

Wang

Hao

et al. 2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Compilers are one of the most fundamental software systems. Compiler testing is important for assuring the quality of compilers. Due to the crucial role of compilers, they have to be well tested. Therefore, automated compiler testing techniques (those based on randomly generated programs) tend to run a large number of test programs (which are test inputs of compilers). The cost for compilation and execution for these test programs is significant. These techniques can take a long period of testing time to detect a relatively small number of compiler bugs. That may cause many practical problems, e.g., bringing a lot of costs including time costs and financial costs, and delaying the development/release cycle. Recently, some approaches have been proposed to accelerate compiler testing by executing test programs that are more likely to trigger compiler bugs earlier according to some criteria. However, these approaches ignore an important aspect in compiler testing: different test programs may have similar test capabilities (i.e., testing similar functionalities of a compiler, even detecting the same compiler bug), which may largely discount their acceleration effectiveness if the test programs with similar test capabilities are executed all the time. Test coverage is a proper approximation to help distinguish them, but collecting coverage dynamically is infeasible in compiler testing since most test programs are generated on the fly by automatic test-generation tools like Csmith. In this paper, we propose the first method to predict test coverage statically for compilers, and then propose to prioritize test programs by clustering them according to the predicted coverage information. The novel approach to accelerating compiler testing through coverage prediction is called COP (short for COverage Prediction). Our evaluation on GCC and LLVM demonstrates that COP significantly accelerates compiler testing, achieving an average of 51.01% speedup in test execution time on an existing dataset including three old release versions of the compilers and achieving an average of 68.74% speedup on a new dataset including 12 latest release versions. Moreover, COP outperforms the state-of-the-art acceleration approach significantly by improving 17.16%∼82.51% speedups in different settings on average.

show abstract

Section: Subjects and Test Programsmentioning

confidence: 99%

Section: Related Work 81 Compiler Testingmentioning

confidence: 99%

Section: Related Work 81 Compiler Testingmentioning

confidence: 99%

See 1 more Smart Citation

Coverage Prediction for Accelerating Compiler Testing

Chen

Wang

Hao

et al. 2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…Hermes inserts code fragments to live regions [53]. Moreover, Proteus applies the EMI technique to test link-time optimizers [34]. The frameworks based on EMI are quite efficient for compiler testing.…”

Section: Related Workmentioning

confidence: 99%

Skeletal program enumeration for rigorous compiler testing

Zhang

Sun

2017

Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation

Self Cite

View full text Add to dashboard Cite

A program can be viewed as a syntactic structure P (syntactic skeleton) parameterized by a collection of identifiers V (variable names). This paper introduces the skeletal program enumeration (SPE) problem: Given a syntactic skeleton P and a set of variables V , enumerate a set of programs P exhibiting all possible variable usage patterns within P. It proposes an effective realization of SPE for systematic, rigorous compiler testing by leveraging three important observations: (1) Programs with different variable usage patterns exhibit diverse control-and data-dependence, and help exploit different compiler optimizations; (2) most real compiler bugs were revealed by small tests (i.e., small-sized P) -this "small-scope" observation opens up SPE for practical compiler validation; and (3) SPE is exhaustive w.r.t. a given syntactic skeleton and variable set, offering a level of guarantee absent from all existing compiler testing techniques.The key challenge of SPE is how to eliminate the enormous amount of equivalent programs w.r.t. α-conversion. Our main technical contribution is a novel algorithm for computing the canonical (and smallest) set of all non-α-equivalent programs. To demonstrate its practical utility, we have applied the SPE technique to test C/C++ compilers using syntactic skeletons derived from their own regression test-suites. Our evaluation results are extremely encouraging. In less than six months, our approach has led to 217 confirmed GCC/Clang bug reports, 119 of which have already been fixed, and the majority are long latent despite extensive prior testing efforts. Our SPE algorithm also provides six orders of magnitude reduction. Moreover, in three weeks, our technique has found 29 CompCert crashing bugs and 42 bugs in two Scala optimizing compilers. These results demonstrate our SPE technique's generality and further illustrate its effectiveness.

show abstract

“…All of them can automatically generate abundant test programs for compilers without undefined behaviors. Simultaneously, various compiler testing techniques have been proposed to mitigate the test oracle problem, such as differential testing [4,22], random testing [21,25], Equivalence Modulo Inputs (EMI) [5], mutation testing [26], and metamorphic testing [27,30]. By employing the above testing techniques, a large number of compiler bugs can be detected.…”

mentioning

confidence: 99%

Compiler testing: a systematic literature analysis

Tang

Zhang

Kong

et al. 2019

Front. Comput. Sci.

View full text Add to dashboard Cite

Compilers are widely-used infrastructures in accelerating the software development, and expected to be trustworthy. In the literature, various testing technologies have been proposed to guarantee the quality of compilers. However, there remains an obstacle to comprehensively characterize and understand compiler testing. To overcome this obstacle, we propose a literature analysis framework to gain insights into the compiler testing area. First, we perform an extensive search to construct a dataset related to compiler testing papers. Then, we conduct a bibliometric analysis to analyze the productive authors, the influential papers, and the frequently tested compilers based on our dataset. Finally, we utilize association rules and collaboration networks to mine the authorships and the communities of interests among researchers and keywords. Some valuable results are reported. We find that the USA is the leading country that contains the most influential researchers and institutions. The most active keyword is "random testing". We also find that most researchers have broad interests within small-scale collaborators in the compiler testing area.

show abstract

Randomized stress-testing of link-time optimizers

Cited by 44 publications

References 26 publications

Coverage Prediction for Accelerating Compiler Testing

Coverage Prediction for Accelerating Compiler Testing

Skeletal program enumeration for rigorous compiler testing

Compiler testing: a systematic literature analysis

Contact Info

Product

Resources

About