From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis

Harman, Mark; O’Hearn, Peter W.

doi:10.1109/scam.2018.00009

Cited by 125 publications

(82 citation statements)

References 126 publications

(155 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is used for modern code review, through which developers submit changes (diffs) and comment on each others' diffs, before they ultimately become accepted into the code base (or are discarded). More than 100,000 diffs are committed to the central repository every week at Facebook, using Phabricator as a central gatekeeper, reporting, curating and testing system [37]. In 2011 Facebook released Phabricator as open source.…”

Section: Phabricatormentioning

confidence: 99%

Deploying Search Based Software Engineering with Sapienz at Facebook

Alshahwan

Gao

Harman

et al. 2018

Search-Based Software Engineering

Self Cite

View full text Add to dashboard Cite

We describe the deployment of the Sapienz Search Based Software Engineering (SBSE) testing system. Sapienz has been deployed in production at Facebook since September 2017 to design test cases, localise and triage crashes to developers and to monitor their fixes. Since then, running in fully continuous integration within Facebook's production development process, Sapienz has been testing Facebook's Android app, which consists of millions of lines of code and is used daily by hundreds of millions of people around the globe. We continue to build on the Sapienz infrastructure, extending it to provide other software engineering services, applying it to other apps and platforms, and hope this will yield further industrial interest in and uptake of SBSE (and hybridisations of SBSE) as a result. This paper was written to accompany the keynote by Mark Harman at the 10 th Symposium on Search-Based Software Engineering (SSBSE 2018), Montpellier September 8-10, 2018. The paper represents the work of all the authors in realising the deployment of search based approaches to large-scale software engineering at Facebook. Author name order is alphabetical; the order is thus not intended to denote any information about the relative contribution of each author.

show abstract

Section: Phabricatormentioning

confidence: 99%

Deploying Search Based Software Engineering with Sapienz at Facebook

Alshahwan

Gao

Harman

et al. 2018

Search-Based Software Engineering

Self Cite

View full text Add to dashboard Cite

show abstract

“…Murphy et al [122] discussed that flaky tests are likely to come up in metamorphic testing whenever floating point calculations are involved. Flaky test detection is a very challenging problem in traditional software testing [270]. It is even more challenging in ML testing because of the oracle problem.…”

Section: Challenges and Opportunitiesmentioning

confidence: 99%

Machine Learning Testing: Survey, Landscapes and Horizons

Zhang

Harman

Ма

et al. 2022

IIEEE Trans. Software Eng.

Self Cite

534

318

View full text Add to dashboard Cite

This paper provides a comprehensive survey of Machine Learning Testing (ML testing) research. It covers 128 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (e.g., the data, learning program, and framework), testing workflow (e.g., test generation and test evaluation), and application scenarios (e.g., autonomous driving, machine translation). The paper also analyses trends concerning datasets, research trends, and research focus, concluding with research challenges and promising research directions in ML testing. Index Terms-machine learning, software testing, deep neural network, ! • Jie M. Zhang and Mark Harman are with CREST, University College London, United Kingdom. Mark Harman is also with Facebook London.

show abstract

“…Such a failure may or may not be spurious. Recently, Harman and O'Hearn [10], reflecting on their experience at Facebook, established the presence of flaky tests as a key challenge in software testing research. In fact, researchers should assume that All Tests Are Flaky (ATAF).…”

Section: B Flaky Tests and Non-deterministic Or Stateful Programsmentioning

confidence: 99%

Assurances in Software Testing: A Roadmap

Böhme

2019

2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER)

View full text Add to dashboard Cite

As researchers, we already understand how to make testing more effective and efficient at finding bugs. However, as fuzzing (i.e., automated testing) becomes more widely adopted in practice, practitioners are asking: Which assurances does a fuzzing campaign provide that exposes no bugs? When is it safe to stop the fuzzer with a reasonable residual risk? How much longer should the fuzzer be run to achieve sufficient coverage?It is time for us to move beyond the innovation of increasingly sophisticated testing techniques, to build a body of knowledge around the explication and quantification of the testing process, and to develop sound methodologies to estimate and extrapolate these quantities with measurable accuracy. In our vision of the future practitioners leverage a rich statistical toolset to assess residual risk, to obtain statistical guarantees, and to analyze the cost-benefit trade-off for ongoing fuzzing campaigns. We propose a general framework as a first starting point to tackle this fundamental challenge and discuss a large number of concrete opportunities for future research. 4 In the quotes, the emphasis in italic letters is mine.

show abstract

From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis

Cited by 125 publications

References 126 publications

Deploying Search Based Software Engineering with Sapienz at Facebook

Deploying Search Based Software Engineering with Sapienz at Facebook

Machine Learning Testing: Survey, Landscapes and Horizons

Assurances in Software Testing: A Roadmap

Contact Info

Product

Resources

About