Test Case Prioritization Based on Information Retrieval Concepts

Kwon, Jung-Hyun; Ko, In-Young; Rothermel, Gregg; Staats, Matt

doi:10.1109/apsec.2014.12

Cited by 21 publications

(49 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Strandberg et al [35] apply a novel prioritization method with multiple factors in a real-world embedded software and show the improvement over industry practice. Other regression test selection techniques have been proposed based on historical test data [16,19,23,25], code dependencies [14], or information retrieval [17,33]. Despite various approaches to test optimization for regression testing, the challenge of applying most of them in practice lies in their complexity and the computational overhead typically required to collect and analyze di erent test parameters needed for prioritization, such as age, test coverage, etc.…”

Section: Related Workmentioning

confidence: 99%

Reinforcement learning for automatic test case prioritization and selection in continuous integration

Spieker

Gotlieb

Marijan

et al. 2017

Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis

197

176

View full text Add to dashboard Cite

Testing in Continuous Integration (CI) involves test case prioritization, selection, and execution at each cycle. Selecting the most promising test cases to detect bugs is hard if there are uncertainties on the impact of committed code changes or, if traceability links between code and tests are not available. This paper introduces R , a new method for automatically learning test case selection and prioritization in CI with the goal to minimize the round-trip time between code commits and developer feedback on failed test cases. The R method uses reinforcement learning to select and prioritize test cases according to their duration, previous last execution and failure history. In a constantly changing environment, where new test cases are created and obsolete test cases are deleted, the R method learns to prioritize error-prone test cases higher under guidance of a reward function and by observing previous CI cycles. By applying R on data extracted from three industrial case studies, we show for the rst time that reinforcement learning enables fruitful automatic adaptive test case selection and prioritization in CI and regression testing. CCS CONCEPTS•Software and its engineering → Software veri cation and validation; Software testing and debugging;

show abstract

Section: Related Workmentioning

confidence: 99%

Reinforcement learning for automatic test case prioritization and selection in continuous integration

Spieker

Gotlieb

Marijan

et al. 2017

Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis

197

176

View full text Add to dashboard Cite

show abstract

“…Gligoric et al (2015) analyse code dependencies to form a regression test suite. Kwon et al (2014) apply information retrieval techniques to the source code and unit tests in order to form a test suite. An entire survey about test case prioritization was conducted by Marijan et al (2013).…”

Section: Related Workmentioning

confidence: 99%

XCSF for Automatic Test Case Prioritization

Rosenbauer

Stein

Pätzel

et al. 2020

Proceedings of the 12th International Joint Conference on Computational Intelligence

View full text Add to dashboard Cite

Testing is a crucial part in the development of a new product. Due to the change from manual testing to automated testing, companies can rely on a higher number of tests. There are certain cases such as smoke tests where the execution of all tests is not feasible and a smaller test suite of critical test cases is necessary. This prioritization problem has just gotten into the focus of reinforcement learning. A neural network and an XCS classifier system have been applied to this task. Another evolutionary machine learning approach is the XCSF which produces, unlike XCS, continuous outputs. In this work we show that XCSF is superior to both the neural network and XCS for this problem.

show abstract

“…Qu et al [111] ordered test cases according to prior fault detection rate using both hand-seeded and mutation faults. Kwon et al [112] proposed a linear regression model to reschedule test cases based on Information Retrieval and coverage information,…”

Section: Attribute Framework Validationmentioning

confidence: 99%

A systematic literature review of how mutation testing supports quality assurance processes

Zhu

Panichella

Zaidman

2018

Software Testing Verif & Rel

View full text Add to dashboard Cite

Mutation testing has been very actively investigated by researchers since the 1970s, and remarkable advances have been achieved in its concepts, theory, technology, and empirical evidence.While the most influential realisations have been summarised by existing literature reviews, we lack insight into how mutation testing is actually applied. Our goal is to identify and classify the main applications of mutation testing and analyse the level of replicability of empirical studies related to mutation testing. To this aim, this paper provides a systematic literature review on the application perspective of mutation testing based on a collection of 191 papers published between 1981 and 2015. In particular, we analysed in which quality assurance processes mutation testing is used, which mutation tools and which mutation operators are employed. Additionally, we also investigated how the inherent core problems of mutation testing, ie, the equivalent mutant problem and the high computational cost, are addressed during the actual usage. The results show that most studies use mutation testing as an assessment tool targeting unit tests, and many of the supporting techniques for making mutation testing applicable in practice are still underdeveloped.Based on our observations, we made 9 recommendations for future work, including an important suggestion on how to report mutation testing in testing experiments in an appropriate manner. KEYWORDSapplication, mutation testing, systematic literature review 1 INTRODUCTION Mutation testing is defined by Jia and Harman [1] as a fault-based testing technique, which provides a testing criterion called the mutation adequacy score. This score can be used to measure the effectiveness of a test set in terms of its ability to detect faults [1]. The principle of mutation testing is to introduce syntactic changes into the original program to generate faulty versions (called mutants) according to well-defined rules (mutation operators) [2]. Mutation testing originated in the 1970s with works from Lipton [3], DeMillo et al. [4], and Hamlet [5] and has been a very active research field over the last few decades. The activeness of the field is in part evidenced by the extensive survey of more than 390 papers on mutation testing that Jia and Harman published in 2011 [1]. Jia and Harman's survey highlights the research achievements that have been made over the years, including the development of tools for a variety of languages and empirical studies performed [1]. Additionally, they highlight some of the actual and inherent problems of mutation testing, among others: (1) the high computational cost caused by generating and executing the numerous mutants and (2) the tremendous time-consuming human investigation required by the test oracle problem and equivalent mutant detection.While existing surveys (eg [1,2,6]) provide us with a great overview of the most influential realisations in research, we lack insight into how mutation testing is actually applied. Specifically, we are interested in analysing ...

show abstract

Test Case Prioritization Based on Information Retrieval Concepts

Cited by 21 publications

References 20 publications

Reinforcement learning for automatic test case prioritization and selection in continuous integration

Reinforcement learning for automatic test case prioritization and selection in continuous integration

XCSF for Automatic Test Case Prioritization

A systematic literature review of how mutation testing supports quality assurance processes

Contact Info

Product

Resources

About