Automated support for classifying software failure reports

Podgurski, Andy; León, David De; Francis, Patrick; Masri, Wes; Minch, M.; Sun, Jiayang; Wang, Bin

doi:10.1109/icse.2003.1201224

Cited by 267 publications

(201 citation statements)

References 20 publications

(13 reference statements)

Supporting

Mentioning

200

Contrasting

Order By: Relevance

“…OTC is related to work on test clustering/filtering/indexing [20,21,22,23,24,25]. Previous work performed clustering based on execution profiles, obtained from monitoring test execution.…”

Section: Related Workmentioning

confidence: 99%

“…Bounded-exhaustive testing can produce a large number of failing tests, and a tester/developer has to map these failures to distinct faults to submit bug reports or debug the code under test. Our technique builds on the ideas from test clustering [20,21,22,23,24,25] where the goal is to split (failing) tests into groups such that all tests in the same group are likely due to the same underlying fault. Previous work mostly considered manually written tests or actual programs runs, and clustering was based on execution profiles obtained from monitoring test execution.…”

Section: Oracle-based Test Clustering (Otc)mentioning

confidence: 99%

See 1 more Smart Citation

Reducing the Costs of Bounded-Exhaustive Testing

Jagannath

Lee

Daniel

et al. 2009

Fundamental Approaches to Software Engineering

View full text Add to dashboard Cite

Abstract. Bounded-exhaustive testing is an automated testing methodology that checks the code under test for all inputs within given bounds: first the user describes a set of test inputs and provides test oracles that check test outputs; then the tool generates all the inputs, executes them on the code under test, and checks the outputs; and finally the user inspects failing tests to submit bug reports. The costs of boundedexhaustive testing include machine time for test generation and execution (which translates into human time waiting for these results) and human time for inspection of results. This paper proposes three techniques that reduce these costs. Sparse Test Generation skips some tests to reduce the time to the first failing test. Structural Test Merging generates a smaller number of larger test inputs (rather than a larger number of smaller test inputs) to reduce test generation and execution time. Oracle-based Test Clustering groups failing tests to reduce the inspection time. Results obtained from the bounded-exhaustive testing of the Eclipse refactoring engine show that these three techniques can substantially reduce the costs while mostly preserving fault-detection capability.

show abstract

“…OTC is related to work on test clustering/filtering/indexing [20,21,22,23,24,25]. Previous work performed clustering based on execution profiles, obtained from monitoring test execution.…”

Section: Related Workmentioning

confidence: 99%

Section: Oracle-based Test Clustering (Otc)mentioning

confidence: 99%

Reducing the Costs of Bounded-Exhaustive Testing

Jagannath

Lee

Daniel

et al. 2009

Fundamental Approaches to Software Engineering

View full text Add to dashboard Cite

show abstract

“…Podgurski et al [22] proposed a semi-automated procedure to classify similar faults and plot them by using a Hierarchical Multi Dimension Scaling (HMDS) algorithm. A tool named Xslice [23] visually differentiates the execution slices of passing and failing part of a test.…”

Section: Related Workmentioning

confidence: 99%

Automated Discovery of Failure Domain

Ahmad¹,

Manuel²

2013

LNSE

View full text Add to dashboard Cite

Abstract-There are several automated random strategies of software testing based on the presence of point, block and strip failure domains inside the whole input domain. As yet no particular, fully automated test strategy has been developed for the discovery and evaluation of the failure domains. We therefore have developed Automated Discovery of Failure Domain (ADFD), a new random test strategy that finds the failures and their domains in a given system under test. It further provides visualization of the identified pass and fail domains. In this paper we describe ADFD strategy, its implementation in York Extensible Testing Infrastructure (YETI) and illustrate its working with the help of an example. We report on experiments in which we tested error-seeded one and two-dimensional numerical programs. Our experimental results show that for each program, ADFD strategy successfully performs identification of failures, failure domains and their representation on graphical chart.

show abstract

“…Francis et al [14] proposed two new tree-based techniques to classify failing executions so that the failing executions resulting from the same cause are grouped together. Podgurski et al [42] proposed to use supervised and unsupervised pattern classification and multivariate visualization to classify failing executions with the related cause. Bowring et al [3] proposed to apply an active learning technique to classify software behavior based on execution data.…”

Section: Application Of Machine Learning In Software Qualitymentioning

confidence: 99%

“…Engineering It is a relatively new topic to apply machine learning to software quality engineering [4,14,18,23,42]. Brun and Ernst [4] isolated fault-revealing properties of a program by applying machine learning to a faulty program and its fixed version.…”

Section: Application Of Machine Learning In Software Qualitymentioning

confidence: 99%

Is This a Bug or an Obsolete Test?

Hao

Lan

Zhang

et al. 2013

ECOOP 2013 – Object-Oriented Programming

View full text Add to dashboard Cite

Abstract. In software evolution, developers typically need to identify whether the failure of a test is due to a bug in the source code under test or the obsoleteness of the test code when they execute a test suite. Only after finding the cause of a failure can developers determine whether to fix the bug or repair the obsolete test. Researchers have proposed several techniques to automate test repair. However, test-repair techniques typically assume that test failures are always due to obsolete tests. Thus, such techniques may not be applicable in real world software evolution when developers do not know whether the failure is due to a bug or an obsolete test. To know whether the cause of a test failure lies in the source code under test or in the test code, we view this problem as a classification problem and propose an automatic approach based on machine learning. Specifically, we target Java software using the JUnit testing framework and collect a set of features that may be related to failures of tests. Using this set of features, we adopt the Best-first Decision Tree Learning algorithm to train a classifier with some existing regression test failures as training instances. Then, we use the classifier to classify future failed tests. Furthermore, we evaluated our approach using two Java programs in three scenarios (within the same version, within different versions of a program, and between different programs), and found that our approach can effectively classify the causes of failed tests.

show abstract

Automated support for classifying software failure reports

Abstract: Abstract

Cited by 267 publications

References 20 publications

Reducing the Costs of Bounded-Exhaustive Testing

Reducing the Costs of Bounded-Exhaustive Testing

Automated Discovery of Failure Domain

Is This a Bug or an Obsolete Test?

Contact Info

Product

Resources

About