Empirical Studies of Test Case Prioritization in a JUnit Testing Environment

Do, Hyunsook; Rothermel, Gregg; Kinneer, Alex

doi:10.1109/issre.2004.18

Cited by 33 publications

(7 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since mutation score is used as an indicator of good quality test cases, using it in test case prioritization gives higher priority to test cases that have a higher chance of exposing faults. Do et al [57] designed and performed an experiment on Java programs that are tested under JUnit framework. They have found that test case prioritization improves the rate of fault detection for JUnit test suites.…”

Section: Related Workmentioning

confidence: 99%

Application of the Law of Minimum and Dissimilarity Analysis to Regression Test Case Prioritization

Ufuktepe

Tuğlular

2023

IEEE Access

View full text Add to dashboard Cite

Regression testing is one of the most expensive processes in testing. Prioritizing test cases in regression testing is critical for the goal of detecting the faults sooner within a large set of test cases. We propose a test case prioritization (TCP) technique for regression testing called LoM-Score inspired by the Law of Minimum (LoM) from biology. This technique calculates the impact probabilities of methods calculated by change impact analysis with forward slicing and orders test cases according to LoM. However, this ordering doesn't consider the possibility that consecutive test cases may be covering the same methods repeatedly. Thereby, such ordering can delay the time of revealing faults that exist in other methods.To solve this problem, we enhance the LoM-Score TCP technique with an adaptive approach, namely with a dissimilarity-based coordinate analysis approach. The dissimilarity-based coordinate analysis uses Jaccard Similarity for calculating the similarity coefficients between test cases in terms of covered methods and the enhanced technique called Dissimilarity-LoM-Score (Dis-LoM-Score) applies a penalty with respective on the ordered test cases. We performed our case study on 10 open-source Java projects from Defects4J, which is a dataset of real bugs and an infrastructure for controlled experiments provided for software engineering researchers. Then, we hand-seeded multiple mutants generated by Major, which is a mutation testing tool. Then we compared our TCP techniques LoM-Score and Dis-LoM-Score with the four traditional TCP techniques based on their Average Percentage of Faults Detected (APFD) results.INDEX TERMS Change impact analysis, regression testing, software testing, test case prioritization.

show abstract

Section: Related Workmentioning

confidence: 99%

Application of the Law of Minimum and Dissimilarity Analysis to Regression Test Case Prioritization

Ufuktepe

Tuğlular

2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Table 2 shows the maximum, mean, and minimum sizes of these test suites. We followed existing work (Do et al 2004;Elbaum et al 2002;Mei et al 2011) to exclude a faulty version from data analyses if more than 20 percent of the test cases detected the fault from the version. The numbers of faulty versions actually used are shown in the rightmost column of Table 1.…”

Section: Benchmark and Test Suitesmentioning

confidence: 99%

“…Regression testing can be more effective by selecting only those test cases relevant to the modified components. Test case prioritization is one of major tasks in regression testing, enabling test cases to be executed in selected order to achieve specific testing purposes, such as a higher fault detection rate (Do et al 2004;Elbaum et al 2002;Srivastava and Thiagarajan 2002). Leung and White (1989) provided a principle of retests by dividing the regression testing problem into two subproblems: test selection and test plan update.…”

Section: Related Workmentioning

confidence: 99%

“…Existing studies on test case prioritization (Do et al 2004;Elbaum et al 2002;Mei et al 2014b), or TCP for short, evaluate various design factors of TCP techniques or these techniques directly based on their effectiveness on average (that is, mean or median effectiveness statistically). Yet, in practice, in a regression test session, a developer only applies at most one test case prioritization technique to one test suite once to test the same version of the same program.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Connecting the Average and the Non-Average

Jia

Mei

Chan

et al. 2015

International Journal of Web Services Research

View full text Add to dashboard Cite

Many existing studies measure the effectiveness of test case prioritization techniques using the average performance on a set of test suites. However, in each regression test session, a real-world developer may only afford to apply one prioritization technique to one test suite to test a service once, even if this application results in an adverse scenario such that the actual performance in this test session is far below the average result achievable by the same technique over the same test suite for the same application. It indicates that assessing the average performance of such a technique cannot provide adequate confidence for developers to apply the technique. The authors ask a couple of questions: To what extent does the effectiveness of prioritization techniques in average scenarios correlate with that in adverse scenarios? Moreover, to what extent may a design factor of this class of techniques affect the effectiveness of prioritization in different types of scenarios? To the best of their knowledge, the authors report in this paper the first controlled experiment to study these two new research questions through more than 300 million APFD and HMFD data points produced from 19 techniques, eight WS-BPEL benchmarks and 1000 test suites prioritized by each technique 1000 times. A main result reveals a strong and linear correlation between the effectiveness in the average scenarios and that in the adverse scenarios. Another interesting result is that many pairs of levels of the same design factors significantly change their relative strengths of being more effective within the same pairs in handling a wide spectrum of prioritized test suites produced by the same techniques over the same test suite in testing the same benchmarks, and the results obtained from the average scenarios are more similar to those of the more effective end than otherwise. This work provides the first piece of strong evidence for the research community to re-assess how they develop and validate their techniques in the average scenarios and beyond.

show abstract

“…A priori knowledge about the characteristics of tests which are more likely to detect a defect is useful for test case prioritisation. In regression testing, test case prioritisation determines which tests should be executed first to potentially improve the defect detection rate [8]. Various prioritisation techniques have been proposed in the literature [20,41], one of which is total function coverage.…”

Section: Introductionmentioning

confidence: 99%

How Effectively Is Defective Code Actually Tested?

Petrić

Hall

Bowes

2018

Proceedings of the 14th International Conference on Predictive Models and Data Analytics in Software Engineering

View full text Add to dashboard Cite

Background: Newspaper headlines still regularly report latent software defects. Such defects have often evaded testing for many years. It remains difficult to identify how well a system has been tested. It also remains difficult to assess how successful at finding defects particular tests are. Coverage and mutation testing are frequently used to asses test effectiveness. We look more deeply at the performance of commonly used JUnit testing by assessing how much JUnit testing was done and how effective that testing was at detecting defects in seven open source systems. Aim: We aim to identify whether defective code has been effectively tested by JUnit tests as non-defective code. We also aim to identify the characteristics of JUnit tests that are related to identifying defects. Methodology: We first extract the defects from seven open source projects using the SZZ algorithm. We match those defects with JUnit tests to identify the proportion of defects that were covered by JUnit tests. We also do the same for non-defective code. We then use Principal Component Analysis and machine learning to investigate the characteristics of JUnit tests that were successful in identifying defects. Results: Our findings suggest that most of the open source systems we investigated are under-tested. On average over 66% of defective methods were not linked to any JUnit tests. We show that the number of methods touched by a JUnit test is strongly related to that test uncovering a defect. Conclusion: More JUnit tests need to be produced for the seven open source systems that we investigate. JUnit tests need to be relatively sophisticated, in particular they should touch more than just one method during the test.

show abstract

Empirical Studies of Test Case Prioritization in a JUnit Testing Environment

Abstract: Abstract

Cited by 33 publications

References 25 publications

Application of the Law of Minimum and Dissimilarity Analysis to Regression Test Case Prioritization

Application of the Law of Minimum and Dissimilarity Analysis to Regression Test Case Prioritization

Connecting the Average and the Non-Average

How Effectively Is Defective Code Actually Tested?

Contact Info

Product

Resources

About