How does regression test prioritization perform in real-world software evolution?

Lu, Yafeng; Lou, Yiling; Cheng, Shiyang; Zhang, Lingming; Hao, Dan; Zhou, Yangfan; Zhang, Lu

doi:10.1145/2884781.2884874

Cited by 96 publications

(128 citation statements)

References 58 publications

Supporting

Mentioning

125

Contrasting

Order By: Relevance

“…This means that test suites from each faulty program version in Defects4J are prioritized for both each faulty version and the latest version (with mutants injected). As Section V describes, this makes for a reasonable comparison due to the fact that past work measuring the impact of software evolution on TCP efficacy has shown that mutation-based performance remains consistent when applying test cases from an earlier program version to both the earlier and later versions [45].…”

Section: B Study Contextmentioning

confidence: 99%

See 1 more Smart Citation

Assessing Test Case Prioritization on Real Faults and Mutants

Luo

Moran

Poshyvanyk

et al. 2018

2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)

View full text Add to dashboard Cite

Test Case Prioritization (TCP) is an important component of regression testing, allowing for earlier detection of faults or helping to reduce testing time and cost. While several TCP approaches exist in the research literature, a growing number of studies have evaluated them against synthetic software defects, called mutants. Hence, it is currently unclear to what extent TCP performance on mutants would be representative of the performance achieved on real faults. To answer this fundamental question, we conduct the first empirical study comparing the performance of TCP techniques applied to both real-world and mutation faults. The context of our study includes eight well-studied TCP approaches, 35k+ mutation faults, and 357 real-world faults from five Java systems in the Defects4J dataset. Our results indicate that the relative performance of the studied TCP techniques on mutants may not strongly correlate with performance on real faults, depending upon attributes of the subject programs. This suggests that, in certain contexts, the best performing technique on a set of mutants may not be the best technique in practice when applied to real faults. We also illustrate that these correlations vary for mutants generated by different operators depending on whether chosen operators reflect typical faults of a subject program. This highlights the importance, particularly for TCP, of developing mutation operators tailored for specific program domains.

show abstract

Section: B Study Contextmentioning

confidence: 99%

“…Thus, it is possible that these results may differ for other TCP evaluation methodologies. However, we chose the experimental methodology set forth in this paper due to the fact that it has been widely used in previous studies [29], [42], [45], [46], [50], [78] and is likely to be used in the future.…”

Section: Threats To Validitymentioning

confidence: 99%

Assessing Test Case Prioritization on Real Faults and Mutants

Luo

Moran

Poshyvanyk

et al. 2018

2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)

View full text Add to dashboard Cite

show abstract

“…During this evolutionary process, it is crucial that developers do not introduce new bugs, known as software regressions. Regression testing is a methodology for efficiently and effectively validating software changes against an existing test suite aimed at detecting such bugs [46], [74]. One of the key tasks of the contemporary practice of continuous regression testing, is test case prioritization (TCP).…”

Section: Introductionmentioning

confidence: 99%

“…Regression test prioritization techniques reorder test executions in order to maximize a certain objective function, such as exposing faults earlier or reducing the execution time cost [46]. This practice can be readily observed in applications to large industrial codebases such as at Microsoft, where researchers have built test prioritization sys- tems for development and maintenance of Windows for a decade [61], [11].…”

Section: Introductionmentioning

confidence: 99%

How Do Static and Dynamic Test Case Prioritization Techniques Perform on Modern Software Systems? An Extensive Study on GitHub Projects

Luo

Moran

Zhang³

et al. 2019

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

Test Case Prioritization (TCP) is an increasingly important regression testing technique for reordering test cases according to a pre-defined goal, particularly as agile practices gain adoption. To better understand these techniques, we perform the first extensive study aimed at empirically evaluating four static TCP techniques, comparing them with state-of-research dynamic TCP techniques across several quality metrics. This study was performed on 58 real-word Java programs encompassing 714 KLoC and results in several notable observations. First, our results across two effectiveness metrics (the Average Percentage of Faults Detected APFD and the cost cognizant APFDc) illustrate that at test-class granularity, these metrics tend to correlate, but this correlation does not hold at test-method granularity. Second, our analysis shows that static techniques can be surprisingly effective, particularly when measured by APFDc. Third, we found that TCP techniques tend to perform better on larger programs, but that program size does not affect comparative performance measures between techniques. Fourth, software evolution does not significantly impact comparative performance results between TCP techniques. Fifth, neither the number nor type of mutants utilized dramatically impact measures of TCP effectiveness under typical experimental settings. Finally, our similarity analysis illustrates that highly prioritized test cases tend to uncover dissimilar faults. and his thesis focused on improving bug reporting for mobile apps through novel applications of program analysis techniques. He has published in several top peerreviewed software engineering venues including: ICSE, ESEC/FSE, IC-SME, ICST, and MSR. He was recently recognized as the second-overall winner among graduate students in the ACM Student Research competition at ESEC/FSE15. Moran is a student member of IEEE and ACM and has served as an external reviewer for ICSE, ICSME, FSE, APSEC, and SCAM. More information available at https://www.kpmoran.com.

show abstract

“…One possible solution to this problem is test case prioritization (TCP). TCP is a topic widely studied in software testing, where test cases are prioritized so that test cases revealing faults are executed earlier, allowing software developers to identify and fix problems earlier [17,25,34,35,39]. A problem with many of the existing TCP techniques is that they rely on coverage information extracted from the source code, such as what branches/statements/functions of the source code are covered by which test cases [4,26,30,40,41].…”

Section: Introductionmentioning

confidence: 99%

TERMINATOR: better automated UI test case prioritization

Fahid

Rothermel

et al. 2019

Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of

View full text Add to dashboard Cite

Automated UI testing is an important component of the continuous integration process of software development. A modern web-based UI is an amalgam of reports from dozens of microservices written by multiple teams. Queries on a page that opens up another will fail if any of that page's microservices fails. As a result, the overall cost for automated UI testing is high since the UI elements cannot be tested in isolation. For example, the entire automated UI testing suite at LexisNexis takes around 30 hours (3-5 hours on the cloud) to execute, which slows down the continuous integration process.To mitigate this problem and give developers faster feedback on their code, test case prioritization techniques are used to reorder the automated UI test cases so that more failures can be detected earlier. Given that much of the automated UI testing is "black box" in nature, very little information (only the test case descriptions and testing results) can be utilized to prioritize these automated UI test cases. Hence, this paper evaluates 17 "black box" test case prioritization approaches that do not rely on source code information. Among these, we propose a novel TCP approach, that dynamically re-prioritizes the test cases when new failures are detected, by applying and adapting a state of the art framework from the total recall problem. Experimental results on LexisNexis automated UI testing data show that our new approach (which we call TERMINATOR), outperformed prior state of the art approaches in terms of failure detection rates with negligible CPU overhead. CCS CONCEPTS• Software and its engineering → Software testing and debugging; • Information systems → Learning to rank.

show abstract

How does regression test prioritization perform in real-world software evolution?

Cited by 96 publications

References 58 publications

Assessing Test Case Prioritization on Real Faults and Mutants

Assessing Test Case Prioritization on Real Faults and Mutants

How Do Static and Dynamic Test Case Prioritization Techniques Perform on Modern Software Systems? An Extensive Study on GitHub Projects

TERMINATOR: better automated UI test case prioritization

Contact Info

Product

Resources

About