A large-scale empirical comparison of static and dynamic test case prioritization techniques

Luo, Qi; Moran, Kevin; Poshyvanyk, Denys

doi:10.1145/2950290.2950344

Cited by 51 publications

(61 citation statements)

References 58 publications

(135 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Third, the T CP add technique does not outperform the T CP tot strategy, contradicting findings from past studies where the T CP add has been shown to perform best overall [21], [46], [62]. Fourth, the results of the Tukey HSD test suggest that for APFD, the performance of the TCP programs does not vary in a statistically significant manner.…”

Section: A Rq 1 : Tcp Effectiveness On Real Faultscontrasting

confidence: 60%

“…While there are many types of existing TCP techniques [16], [23], [29], [38], [66], [80], one common dichotomous classification, static [31], [42], [81] and dynamic techniques [18], [19], [22], [41], [53], [65], [72], [79], relates to the type of information used to perform the prioritization. Static approaches utilize information extracted from source and test code and dynamic techniques rely on information collected at runtime (e.g., coverage information per test case) to prioritize test cases [46]. Additional classifications exist, such as the distinction between white-box and black-box techniques [30].…”

Section: A Tcp Problem Formulationmentioning

confidence: 99%

“…Conversely, those that only require program input or output information are classified as black-box techniques. Approaches that only require testcode have been classified as black-box [30], [46], however, in this paper we more accurately refer to these techniques as grey-box since they require access to test code with references to the underlying program. There are also other approaches [32], [63], [70] that use other types of information to perform the prioritization, such as code-changes and requirements, that do not fall neatly into these categories.…”

Section: A Tcp Problem Formulationmentioning

confidence: 99%

“…To collect coverage information, we used the ASM bytecode manipulation and analysis toolset [4]. In our empirical study, we chose to use statement-level coverage information, as this allows for optimal performance of TCP techniques [46]. Furthermore, we utilize JDT [25] to extract textual information for each test method, which is used by string-based and topic-based approaches.…”

Section: Experiments Tools and Hardwarementioning

confidence: 99%

See 3 more Smart Citations

Assessing Test Case Prioritization on Real Faults and Mutants

Luo

Moran

Poshyvanyk

et al. 2018

2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)

Self Cite

View full text Add to dashboard Cite

Test Case Prioritization (TCP) is an important component of regression testing, allowing for earlier detection of faults or helping to reduce testing time and cost. While several TCP approaches exist in the research literature, a growing number of studies have evaluated them against synthetic software defects, called mutants. Hence, it is currently unclear to what extent TCP performance on mutants would be representative of the performance achieved on real faults. To answer this fundamental question, we conduct the first empirical study comparing the performance of TCP techniques applied to both real-world and mutation faults. The context of our study includes eight well-studied TCP approaches, 35k+ mutation faults, and 357 real-world faults from five Java systems in the Defects4J dataset. Our results indicate that the relative performance of the studied TCP techniques on mutants may not strongly correlate with performance on real faults, depending upon attributes of the subject programs. This suggests that, in certain contexts, the best performing technique on a set of mutants may not be the best technique in practice when applied to real faults. We also illustrate that these correlations vary for mutants generated by different operators depending on whether chosen operators reflect typical faults of a subject program. This highlights the importance, particularly for TCP, of developing mutation operators tailored for specific program domains.

show abstract

Section: A Rq 1 : Tcp Effectiveness On Real Faultscontrasting

confidence: 60%

Section: A Tcp Problem Formulationmentioning

confidence: 99%

Section: A Tcp Problem Formulationmentioning

confidence: 99%

Section: Experiments Tools and Hardwarementioning

confidence: 99%

See 2 more Smart Citations

Assessing Test Case Prioritization on Real Faults and Mutants

Luo

Moran

Poshyvanyk

et al. 2018

2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)

Self Cite

View full text Add to dashboard Cite

show abstract

“…This approach has been successfully used for bug localization in the BLUiR system before [18]. Static test prioritization operates without change data and tries to spread similar tests to cover many concepts (diversity) or most of the program (coverage) up-front [10]. Topic models have been used by Thomas, Hemmati, Hassan, and Blostein [21] to model these concepts as topics over lexical features and hence address the concept coverage problem from a lexical viewpoint.…”

Section: :21mentioning

confidence: 99%

Lightweight Lexical Test Prioritization for Immediate Feedback

Mattis

Hirschfeld

2020

Programming

View full text Add to dashboard Cite

The practice of unit testing enables programmers to obtain automated feedback on whether a currently edited program is consistent with the expectations specified in test cases. Feedback is most valuable when it happens immediately, as defects can be corrected instantly before they become harder to fix. With growing and longer running test suites, however, feedback is obtained less frequently and lags behind program changes.The objective of test prioritization is to rank tests so that defects, if present, are found either as early as possible or with the least costs. While there are numerous static approaches that output a ranking of tests solely based on the current version of a program, we focus on change-based test prioritization, which recommends tests that likely fail in response to the most recent program change. The canonical approach relies on coverage data and prioritizes tests that cover the changed region, but obtaining and updating coverage data is costly. More recently, information retrieval techniques that exploit overlapping vocabulary between change and tests have proven to be powerful, yet lightweight.In this work, we demonstrate the capabilities of information retrieval for prioritizing tests in dynamic programming languages using Python as example. We discuss and measure previously understudied variation points, including how contextual information around a program change can be used, and design alternatives to the widespread TF-IDF retrieval model tailored to retrieving failing tests.To obtain program changes with associated test failures, we designed a tool that generates a large set of faulty changes from version history along with their test results. Using this data set, we compared existing and new lexical prioritization strategies using four open-source Python projects, showing large improvements over untreated and random test orders and results consistent with related work in statically typed languages.We conclude that lightweight IR-based prioritization strategies are effective tools to predict failing tests in the absence of coverage data or when static analysis is intractable like in dynamic languages. This knowledge can benefit both individual programmers that rely on fast feedback, as well as operators of continuous integration infrastructure, where resources can be freed sooner by detecting defects earlier in the build cycle. ACM CCS 2012

show abstract

Semantic‐aware two‐phase test case prioritization for continuous integration

Li,

Wang,

Wang

et al. 2023

Software Testing Verif & Rel

View full text Add to dashboard Cite

SummaryContinuous integration (CI) is a widely applied development practice to allow frequent integration of software changes, detecting early faults. However, extremely frequent builds consume amounts of time and resources in such a scenario. It is quite challenging for existing test case prioritization (TCP) to address this issue due to the time‐consuming information collection (e.g. test coverage) or inaccurately modelling code semantics to result in the unsatisfied prioritization. In this paper, we propose a semantic‐aware two‐phase TCP framework, named SatTCP, which combines the coarse‐grained filtering and fine‐grained prioritization to perform the precise TCP with low time costs for CI. It consists of three parts: (1) code representation, parsing the programme changes and test cases to obtain the code change and test case representations; (2) coarse‐grained filtering, conducting the preliminary ranking and filtering of test cases based on information retrieval; and (3) fine‐grained prioritization, training a pretrained Siamese language model based on the filtered test set to further sort the test cases via semantic similarity. We evaluate SatTCP on a large‐scale, real‐world dataset with cross‐project validation from fault detection efficiency and time costs and compare it with five baselines. The results show that SatTCP outperforms all baselines by 6.3%–45.6% for mean average percentage of fault detected per cost (APFDc), representing an obvious upward trend as the project scale increases. Meanwhile, SatTCP can reduce the real CI testing by 71.4%, outperforming the best baseline by 17.2% for time costs on average. Furthermore, we discuss the impact of different configurations, flaky tests and hybrid techniques on the performance of SatTCP, respectively.

show abstract

A large-scale empirical comparison of static and dynamic test case prioritization techniques

Cited by 51 publications

References 58 publications

Assessing Test Case Prioritization on Real Faults and Mutants

Assessing Test Case Prioritization on Real Faults and Mutants

Lightweight Lexical Test Prioritization for Immediate Feedback

Semantic‐aware two‐phase test case prioritization for continuous integration

Contact Info

Product

Resources

About