Making System User Interactive Tests Repeatable: When and What Should We Control?

Gao, Zebao; Liang, Yalan; Cohen, Myra B.; Memon, Atif M.; Wang, Zhen

doi:10.1109/icse.2015.28

Cited by 30 publications

(12 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several authors have commented on the reasons for this flakiness [28,37,46,56,62], finding that one of the most common causes lies in the prevalent use of asynchronous waits; functions make asynchronous calls to services, but may not wait sufficient time for those services to respond, producing a different result to that had they waited slightly longer.…”

Section: Crash Reproduction Workflowmentioning

confidence: 99%

Deploying Search Based Software Engineering with Sapienz at Facebook

Alshahwan

Gao

Harman

et al. 2018

Search-Based Software Engineering

View full text Add to dashboard Cite

We describe the deployment of the Sapienz Search Based Software Engineering (SBSE) testing system. Sapienz has been deployed in production at Facebook since September 2017 to design test cases, localise and triage crashes to developers and to monitor their fixes. Since then, running in fully continuous integration within Facebook's production development process, Sapienz has been testing Facebook's Android app, which consists of millions of lines of code and is used daily by hundreds of millions of people around the globe. We continue to build on the Sapienz infrastructure, extending it to provide other software engineering services, applying it to other apps and platforms, and hope this will yield further industrial interest in and uptake of SBSE (and hybridisations of SBSE) as a result. This paper was written to accompany the keynote by Mark Harman at the 10 th Symposium on Search-Based Software Engineering (SSBSE 2018), Montpellier September 8-10, 2018. The paper represents the work of all the authors in realising the deployment of search based approaches to large-scale software engineering at Facebook. Author name order is alphabetical; the order is thus not intended to denote any information about the relative contribution of each author.

show abstract

Section: Crash Reproduction Workflowmentioning

confidence: 99%

Deploying Search Based Software Engineering with Sapienz at Facebook

Alshahwan

Gao

Harman

et al. 2018

Search-Based Software Engineering

View full text Add to dashboard Cite

show abstract

“…An important difference between verification and testing derives from the way in which execution has increasingly become nondeterministic. Whereas programming languages have included new semantic features that have posed novel challenges for verification research and practice, the dramatic increase in the stochastic behaviour of most deployed systems, over a similar period, has posed challenges for software testing; tests are 'flaky' [50], [87], [100]. In this section we explore this test-specific challenge, which can be summarised with the aphorism: Assume all Tests Are Flaky (ATAF).…”

Section: Survive and Thrive Even When We Assume Tests Are Flaky mentioning

confidence: 99%

“…Gao et al [50] consider the related problem of understanding the impact of flakiness at different layers (user interaction layer, behavioural layer, and code layer), concluding that testers should do their best to control flakiness at each level.…”

Section: A Prior Work On the Flakiness Problemmentioning

confidence: 99%

From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis

Harman

O’Hearn

2018

2018 IEEE 18th International Working Conference on Source Code Analysis and Manipulation (SCAM)

125

View full text Add to dashboard Cite

This paper 1 describes some of the challenges and opportunities when deploying static and dynamic analysis at scale, drawing on the authors' experience with the Infer and Sapienz Technologies at Facebook, each of which started life as a research-led start-up that was subsequently deployed at scale, impacting billions of people worldwide.The paper identifies open problems that have yet to receive significant attention from the scientific community, yet which have potential for profound real world impact, formulating these as research questions that, we believe, are ripe for exploration and that would make excellent topics for research projects.

show abstract

“…In other words, for an identical system-call with the same input parameters, several possible execution paths may exist. Conse-quently, it is no longer feasible to assess the correctness of this type of systems with a unique iteration of a testcase for each combination of input parameters [30]. Besides, it is challenging (if feasible) to force the execution of a specific path.…”

Section: Problem Statementmentioning

confidence: 99%

Statistical Test Coverage for Linux-Based Next-Generation Autonomous Safety-Related Systems

Allende

Guire²,

Pérez-Cerrolaza

et al. 2021

IEEE Access

View full text Add to dashboard Cite

Autonomous systems represent a significant leap forward in the ongoing technological evolution of dependable and safety-related systems, integrating features such as artificial intelligence, high-performance computing devices, General Purpose Operating Systems (GPOS) (e.g., GNU/Linux) and security requirements. Nonetheless, traditionally employed safety techniques and measures were not defined for safety-related systems with such features. Consequently, the need to research new methods and measures emerges in order to be able to achieve appropriate safety assurance. In this manuscript, we explore the limitations of traditional test coverage techniques, and we provide two complementary methods to pave the way towards the testing of Linux-based complex safety-related systems. The methods are presented and applied to a Linux-based Autonomous Emergency Braking (AEB) case study, specifically focusing on the kernel execution path test coverage.

show abstract

Making System User Interactive Tests Repeatable: When and What Should We Control?

Cited by 30 publications

References 29 publications

Deploying Search Based Software Engineering with Sapienz at Facebook

Deploying Search Based Software Engineering with Sapienz at Facebook

From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis

Statistical Test Coverage for Linux-Based Next-Generation Autonomous Safety-Related Systems

Contact Info

Product

Resources

About