Taming Google-scale continuous testing

Memon, Atif M.; Gao, Zebao; Nguyen, Bao N.; Dhanda, Sanjeev; Nickell, Eric; Siemborski, Rob; Micco, John

doi:10.1109/icse-seip.2017.16

Cited by 182 publications

(93 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Deployability. Section XI: Our experience lies primary in the challenges of deployment within continuous integration environments, which are increasingly industrially prevalent [101]. In Section XI we describe some of the lessons we learned in our efforts to deploy research on testing and verification at Facebook, focusing on those lessons that we believe to be generic to all continuous integration environments, but illustrating with specifics from Facebook's deployment of Infer and Sapienz.…”

Section: Introductionmentioning

confidence: 99%

From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis

Harman

O’Hearn

2018

2018 IEEE 18th International Working Conference on Source Code Analysis and Manipulation (SCAM)

125

View full text Add to dashboard Cite

This paper 1 describes some of the challenges and opportunities when deploying static and dynamic analysis at scale, drawing on the authors' experience with the Infer and Sapienz Technologies at Facebook, each of which started life as a research-led start-up that was subsequently deployed at scale, impacting billions of people worldwide.The paper identifies open problems that have yet to receive significant attention from the scientific community, yet which have potential for profound real world impact, formulating these as research questions that, we believe, are ripe for exploration and that would make excellent topics for research projects.

show abstract

Section: Introductionmentioning

confidence: 99%

From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis

Harman

O’Hearn

2018

2018 IEEE 18th International Working Conference on Source Code Analysis and Manipulation (SCAM)

125

View full text Add to dashboard Cite

show abstract

“…A number of observations based on the presented results of the experiments lead to interesting conclusions. 1]. This verifies that the model trained on de-flaked data is not worse at "catching" failed tests than those that flaked, a desired behavior.…”

Section: Impact Of Test Flakinessmentioning

confidence: 58%

“…While it is possible to base test selection on dynamic analysis of the code a particular change is based on, one can no longer make strict guarantees on quality of this approach, as even a small code change can arbitrarily alter runtime behavior. Also, maintaining per-test code coverage information accurate enough to drive the test selection process is impractical in large monolithic repositories [1], while recording it requires language-specific infrastructure and is challenging across language boundaries.…”

Section: Introductionmentioning

confidence: 99%

Predictive Test Selection

Machalica

Samylkin

Porth

et al. 2019

2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)

View full text Add to dashboard Cite

Change-based testing is a key component of continuous integration at Facebook. However, a large number of tests coupled with a high rate of changes committed to our monolithic repository make it infeasible to run all potentiallyimpacted tests on each change. We propose a new predictive test selection strategy which selects a subset of tests to exercise for each change submitted to the continuous integration system. The strategy is learned from a large dataset of historical test outcomes using basic machine learning techniques. Deployed in production, the strategy reduces the total infrastructure cost of testing code changes by a factor of two, while guaranteeing that over 95% of individual test failures and over 99.9% of faulty changes are still reported back to developers. The method we present here also accounts for the non-determinism of test outcomes, also known as test flakiness.

show abstract

“…$15.00 https://doi.org/10.1145/nnnnnnn.nnnnnnn runs. " [23]. By adopting high quality software testing in the continuous integration context, software companies are now releasing software much more frequently.…”

Section: Introductionmentioning

confidence: 99%

Goal-oriented mutation testing with focal methods

Vercammen

Ghafari

Demeyer

et al. 2018

Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation

View full text Add to dashboard Cite

Mutation testing is the state-of-the-art technique for assessing the fault-detection capacity of a test suite. Unfortunately, mutation testing consumes enormous computing resources because it runs the whole test suite for each and every injected mutant. In this paper we explore fine-grained traceability links at method level (named focal methods), to reduce the execution time of mutation testing and to verify the quality of the test cases for each individual method, instead of the usually verified overall test suite quality. Validation of our approach on the open source Apache Ant project shows a speed-up of 573.5x for the mutants located in focal methods with a quality score of 80

show abstract

Taming Google-scale continuous testing

Cited by 182 publications

References 24 publications

From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis

From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis

Predictive Test Selection

Goal-oriented mutation testing with focal methods

Contact Info

Product

Resources

About