Efficient time-aware prioritization with knapsack solvers

Alspaugh, Sara; Walcott, Kristen R.; Belanich, Michael; Kapfhammer, Gregory M.; Soffa, Mary Lou

doi:10.1145/1353673.1353676

Cited by 32 publications

(27 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Some research effort has been spent to deal with the test execution time, adopting for example a genetic algorithm to reorder test suites in light of testing time constraints (under the assumption that all the tests have the same execution time) (Walcott et al 2006). Other approaches are integer linear programming (Zhang et al 2009) and knapsack solvers (Alspaugh et al 2007). The effect of constraints (on time to setup testing, time to identify and repair obsolete tests, and human time to inspect results) on test prioritization has been analyzed in Do et al (2010) and Do and Rothermel (2006).…”

Section: Previous Workmentioning

confidence: 99%

Similarity-based prioritization of test case automation

et al. 2018

View full text Add to dashboard Cite

The importance of efficient software testing procedures is driven by an ever increasing system complexity as well as global competition. In the particular case of manual test cases at the system integration level, where thousands of test cases may be executed before release, time must be well spent in order to test the system as completely and as efficiently as possible. Automating a subset of the manual test cases, i.e, translating the manual instructions to automatically executable code, is one way of decreasing the test effort. It is further common that test cases exhibit similarities, which can be exploited through reuse when automating a test suite. In this paper, we investigate the potential for reducing test effort by ordering the test cases before such automation, given that we can reuse already automated parts of test cases. In our analysis, we investigate several approaches for prioritization in a case study at a large Swedish vehicular manufacturer. The study analyzes the effects with respect to test effort, on four projects with a total of 3919 integration test cases constituting 35,180 test steps, written in natural language. The results show that for the four projects considered, the difference in expected manual effort between the best and the worst order found is on average 12 percentage points. The results also show that our proposed prioritization method is nearly as good as more resource demanding meta-heuristic approaches Software Qual J at a fraction of the computational time. Based on our results, we conclude that the order of automation is important when the set of test cases contain similar steps (instructions) that cannot be removed, but are possible to reuse. More precisely, the order is important with respect to how quickly the manual test execution effort decreases for a set of test cases that are being automated.

show abstract

Section: Previous Workmentioning

confidence: 99%

Similarity-based prioritization of test case automation

et al. 2018

View full text Add to dashboard Cite

show abstract

“…We obtained code change information using sandmark [7] as a byte code differencing tool, and we obtained software quality metrics (including data on coupling, cohesion, and complexity) from the Chidamber-Kemerer metrics suite [6] using the ckjm program. 1 Using this information and the original test suite, prioritization techniques produce reordered test suites for each version of P .…”

Section: Relative Cost-benefitmentioning

confidence: 99%

“…Because our focus is regression testing and detection of regression faults, in applying the model we considered only files that have been changed from the previous version. Using the LOC model, we obtained various ranges of numbers of faults across the different versions of our object programs; these ranges were: ant (3-39), jmeter (12)(13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25)(26), xml-security (5-10), nanoxml (1)(2)(3)(4)(5), and galileo (1)(2)(3)(4)(5)(6)(7)(8). Based on these numbers, for each version of each program we randomly selected several mutant groups of sizes falling within those ranges from the set of that version's mutation faults.…”

Section: Experiments Setupmentioning

confidence: 99%

See 1 more Smart Citation

The Effects of Time Constraints on Test Case Prioritization: A Series of Controlled Experiments

Mirarab

Tahvildari

et al. 2010

IIEEE Trans. Software Eng.

140

View full text Add to dashboard Cite

Regression testing is an expensive process used to validate modified software. Test case prioritization techniques improve the cost-effectiveness of regression testing by ordering test cases such that those that are more important are run earlier in the testing process. Many prioritization techniques have been proposed and evidence shows that they can be beneficial. It has been suggested, however, that the time constraints that can be imposed on regression testing by various software development processes can strongly affect the behavior of prioritization techniques. If this is correct, a better understanding of the effects of time constraints could lead to improved prioritization techniques, and improved maintenance and testing processes. We therefore conducted a series of experiments to assess the effects of time constraints on the costs and benefits of prioritization techniques. Our first experiment manipulates time constraint levels and shows that time constraints do play a significant role in determining both the cost-effectiveness of prioritization and the relative cost-benefit tradeoffs among techniques. Our second experiment replicates the first experiment, controlling for several threats to validity including numbers of faults present, and shows that the results generalize to this wider context. Our third experiment manipulates the numbers of faults present in programs to examine the effects of faultiness levels on prioritization, and shows that faultiness level affects the relative costeffectiveness of prioritization techniques. Taken together, these results have several implications for test engineers wishing to cost-effectively regression test their software systems. These include suggestions about when and when not to prioritize, what techniques to employ, and how differences in testing processes may relate to prioritization cost-effectiveness.

show abstract

“…An evaluation and discussion of the tradeoffs between time-aware test case selection quality and the efficiency of selecting using 0/1 Knapsack approximation algorithms [9,116].…”

Section: Contributions Of the Dissertationmentioning

confidence: 99%

Testing in Resource-Constrained Environments

Walcott-Justice¹

View full text Add to dashboard Cite

In recent software development environments, resources including time, memory, and power constrain the execution of software test cases. Often the cost of running all test cases is prohibitively expensive, necessitating techniques to select test cases of the highest quality for test execution. While many test case selection and prioritization techniques exist to select test cases that are more likely to find faults, none take the resource constraints under which the tests will operate directly into consideration. Another cost in test case execution comes from structural testing. Structural testing is an important technique in which during execution test cases are also evaluated for structural coverage. This is typically performed through the use of code instrumentation. However, instrumentation incurs high overhead in terms of time and code growth, making it difficult to use in resource-constrained environments such as mobile device testing.This thesis develops and evaluates techniques for efficient and effective execution of test cases in resource-constrained environments. The first set of techniques selects high quality test cases from test suites while taking resource constraints into account. The resulting test selection is guaranteed to execute within the specified resource constraints, and the resulting test cases have the highest possible potential for overall fault detection during their constrained execution. The selected tests are additionally ordered in a way that is likely to find faults earlier in test execution rather than later. These techniques represent the first to guarantee that test selections will execute within a given constraint, and they offer a spectrum of options for selecting tests to execute in resource-constrained environments.The second set of techniques evaluates the quality of test cases during execution withi Abstract ii out the need of expensive and intrusive instrumentation. Instead, our quality evaluation techniques exploit recent hardware advances to monitor program execution. By leveraging these hardware advances, up to 90% of branch coverage and 79% of statement coverage can be determined with less time overhead compared to instrumentation. Because the techniques require little or no code modification or dynamic code insertion and can run on commodity machines, tablets, or mobile devices, the techniques also enable test case execution and evaluation on resource-constrained devices. No specialized hardware or components are required. The techniques are combined into a runtime and static testing system called THeME: Testing by Hardware Monitoring Events. THeME is a portable, extensible system, making it applicable for use in a wide range of resource-constrained environments.

show abstract

Efficient time-aware prioritization with knapsack solvers

Cited by 32 publications

References 11 publications

Similarity-based prioritization of test case automation

Similarity-based prioritization of test case automation

The Effects of Time Constraints on Test Case Prioritization: A Series of Controlled Experiments

Testing in Resource-Constrained Environments

Contact Info

Product

Resources

About