Taming compiler fuzzers

Yang, Chen; Groce, Alex; Zhang, Chaoqiang; Wong, W. Eric; Fern, Xiaoli Z.; Eide, Eric; Regehr, John

doi:10.1145/2491956.2462173

Cited by 121 publications

(58 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…CCG is a random C program generator that focuses on finding compiler crashing bugs [3]. Csmith is another C program generator that can find both crashing and miscompilation bugs [4,19,25]. Based on the idea of differential testing [12], Csmith randomly generates C programs and checks for deviant behavior across compilers or compiler versions.…”

Section: Related Workmentioning

confidence: 99%

Randomized stress-testing of link-time optimizers

Sun

2015

Proceedings of the 2015 International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

Link-time optimization (LTO) is an increasingly important and adopted modern optimization technology. It is currently supported by many production compilers, including GCC, LLVM, and Microsoft Visual C/C++. Despite its complexity, but because it is more recent, LTO is relatively less tested compared to the more mature, traditional optimizations. To evaluate and help improve the quality of LTO, we present the first extensive effort to stress-test the LTO components of GCC and LLVM, the two most widely-used production C compilers. In 11 months, we have discovered and reported 37 bugs (12 in GCC; 25 in LLVM). Developers have confirmed 21 of our bugs, and fixed 11 of them.Our core technique is differential testing and realized in the tool Proteus. We leverage existing compiler testing tools (Csmith and Orion) to generate single-file test programs and address two important challenges specific for LTO testing. First, to thoroughly exercise LTO, Proteus automatically transforms a single-file program into multiple compilation units and stochastically assigns each an optimization level. Second, for effective bug reporting, we develop a practical mechanism to reduce LTO bugs involving multiple files. Our results clearly demonstrate Proteus's utility; we plan to make ours a continuous effort in validating link-time optimizers.

show abstract

Section: Related Workmentioning

confidence: 99%

Randomized stress-testing of link-time optimizers

Sun

2015

Proceedings of the 2015 International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

show abstract

“…Moreover, more specialized versions of the SCH in domains where large-scale automated testing is frequently applied (e.g. compilers [11] or systems utilities [45]) may be useful to practitioners and researchers in these areas.…”

Section: How Can Users Of Code Coverage Sleepmentioning

confidence: 99%

“…Each suite is produced by running jsfunfuzz for 30 minutes, removing the possibility of suite size (if measured using computational effort, which seems the most reasonable way to measure it) as a confounding factor. The number of faults detected is estimated (with relatively good accuracy, we believe) using a binary search through the source code repository to find the change that "fixes" each failing test [11,29].…”

Section: A Sketch Of a Data Setmentioning

confidence: 99%

“…for full, functional correctness properties -the easiest way to determine if a file system functions properly is to test against another, correct, file system); in some cases, the oracle cannot be automated, and requires human intervention [30]. Even with a perfect oracle, determining the list of faults from a set of failed tests is a very hard problem [11].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Coverage and Its Discontents

Groce

Alipour

Gopinath

2014

Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming &Amp; Software

Self Cite

View full text Add to dashboard Cite

Everyone wants to know one thing about a test suite: will it detect enough bugs? Unfortunately, in most settings that matter, answering this question directly is impractical or impossible. Software engineers and researchers therefore tend to rely on various measures of code coverage (where mutation testing is considered a form of syntactic coverage). A long line of academic research efforts have attempted to determine whether relying on coverage as a substitute for fault detection is a reasonable solution to the problems of test suite evaluation. This essay argues that the profusion of coverage-related literature is in part a sign of an underlying uncertainty as to what exactly it is that measuring coverage should achieve, as well as how we would know if it can, in fact, achieve it. We propose some solutions and mitigations, but the primary focus of this essay is to clarify the state of current confusions regarding this key problem for effective software testing.

show abstract

“…Examining the code, there does not appear to be any link between the feature and the failures other than that the feature causes js to fail in one of the three "common" ways before the less frequent failures (all requiring longer executions on average) can take place. Complex systems will almost always have multiple faults that differ in frequency of appearance; in our experience, for compilers and JavaScript engines, failure rates for different faults typically exhibit a power law curve and sometimes a "double power law" curve [23]. Masking may therefore merit special attention as a source of suppression.…”

Section: E Subject Details: Causes Of Suppressionmentioning

confidence: 99%

Help, help, i'm being suppressed! The significance of suppressors in software testing

Groce

Zhang

Alipour

et al. 2013

2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE)

Self Cite

View full text Add to dashboard Cite

Test features are basic compositional units used to describe what a test does (and does not) involve. For example, in API-based testing, the most obvious features are function calls; in grammar-based testing, the obvious features are the elements of the grammar. The relationship between features as abstractions of tests and produced behaviors of the tested program is surprisingly poorly understood. This paper shows how large-scale random testing modified to use diverse feature sets can uncover causal relationships between what a test contains and what the program being tested does. We introduce a general notion of observable behaviors as targets, where a target can be a detected fault, an executed branch or statement, or a complex coverage entity such as a state, predicate-valuation, or program path. While it is obvious that targets have triggers -features without which they cannot be hit by a testthe notion of suppressors -features which make a test less likely to hit a target -has received little attention despite having important implications for automated test generation and program understanding. For a set of subjects including C compilers, a flash file system, and JavaScript engines, we show that suppression is both common and important.

show abstract

Taming compiler fuzzers

Cited by 121 publications

References 44 publications

Randomized stress-testing of link-time optimizers

Randomized stress-testing of link-time optimizers

Coverage and Its Discontents

Help, help, i'm being suppressed! The significance of suppressors in software testing

Contact Info

Product

Resources

About

Taming compiler fuzzers

Cited by 121 publications

References 44 publications

Randomized stress-testing of link-time optimizers

Randomized stress-testing of link-time optimizers

Coverage and Its Discontents

Help, help, i'm being suppressed&#x0021; The significance of suppressors in software testing

Contact Info

Product

Resources

About

Help, help, i'm being suppressed! The significance of suppressors in software testing