How to Detect Performance Changes in Software History

Reichelt, David Georg; Kühne, Stefan

doi:10.1145/3185768.3186404

Cited by 9 publications

(5 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To mitigate this threat, we have compared 30 executions of each test case on the original program and each mutant.Additionally, we have performed a statistical analysis of the execution times and we have used three different thresholds (p-values). More complex methods could be used, like the ones proposed by Reichelt and Kühne [9]. However, the methods that require many executions would make the application of the technique excessively expensive.…”

Section: Construct Validitymentioning

confidence: 99%

“…In contrast to functional bugs, performance bugs do not usually produce wrong results or crashes in the program under test, and therefore, they cannot be detected by simply inspecting the program output. Furthermore, performance indicators are by nature non-deterministic and can vary among executions due to numerous factors, such as the device hardware, configuration settings or the current workload [9]. For example, suppose a mobile shopping app that consumes 30 Mb of memory: Is this the expected performance?…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Performance mutation testing

Delgado‐Pérez

Sánchez

Segura

et al. 2020

Software Testing Verif & Rel

View full text Add to dashboard Cite

Performance bugs are known to be a major threat to the success of software products. Performance tests aim to detect performance bugs by executing the program through test cases and checking whether it exhibits a noticeable performance degradation. The principles of mutation testing, a well-established testing technique for the assessment of test suites through the injection of artificial faults, could be exploited to evaluate and improve the detection power of performance tests. However, the application of mutation testing to assess performance tests, henceforth called performance mutation testing (PMT), is a novel research topic with numerous open challenges. In previous papers, we identified some key challenges related to PMT. In this work, we go a step further and explore the feasibility of applying PMT at the source-code level in generalpurpose languages. To do so, we revisit concepts associated with classical mutation testing, and design seven novel mutation operators to model known bug-inducing patterns. As a proof of concept, we applied traditional mutation operators as well as performance mutation operators to open-source C++ programs. The results reveal the potential of the new performance-mutants to help assess and enhance performance tests when compared to traditional mutants. A review of live mutants in these programs suggests that they can induce the design of special test inputs. In addition to these promising results, our work brings a whole new set of challenges related to PMT, which will hopefully serve as a starting point for new contributions in the area.

show abstract

Section: Construct Validitymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Performance mutation testing

Delgado‐Pérez

Sánchez

Segura

et al. 2020

Software Testing Verif & Rel

View full text Add to dashboard Cite

show abstract

“…Some approaches to enhance performance testing include the use of static code analysis to discover performance errors. The authors of [18] determine a test plan based on code commits and the use of artificial unit tests to compare the performance between different versions. Meanwhile, the work presented on [17] describes the performance analysis of software systems as a comparison of two versions of software and their performance results to find possible (regression) bugs.…”

Section: Background and Related Workmentioning

confidence: 99%

DYNAMOJM: A JMeter Tool for Performance Testing Using Dynamic Workload Adaptation

Huerta-Guevara

Ayala-Rivera

Murphy

et al. 2019

Testing Software and Systems

View full text Add to dashboard Cite

Performance testing is a critical task to assure optimal experience for users, especially when there are high loads of concurrent users. JMeter is one of the most widely used tools for load and stress testing. With JMeter, it is possible to test the performance of static and dynamic resources on the web. This paper presents DYNAMOJM, a novel tool built on top of JMeter that enables testers to create a dynamic workload for performance testing. This tool implements the DYNAMO approach, which has proven useful to find performance issues more efficiently than static testing techniques.

show abstract

“…Other performance testing approaches are based on static code analysis. For instance, the authors of [34] determine the performance tests based on commits and the usage of unittests. Meanwhile, the work presented in [33] describes the performance analysis of software systems as a comparison of two versions of software and their performance results to find possible (regression) bugs.…”

Section: Background and Related Workmentioning

confidence: 99%

“…Conducting performance testing typically requires a lot of effort and expertise from testers. The performance of each application is unique and although the application could be the same, different versions and releases are still distinctive, so the set of performance tests needs to be updated accordingly [34]. This problem emphasizes the need for an expert on the application under test in order to get some insights regarding its weakness [38].…”

Section: Introductionmentioning

confidence: 99%

Towards an Efficient Performance Testing Through Dynamic Workload Adaptation

Huerta-Guevara

Ayala-Rivera

Murphy

et al. 2019

Testing Software and Systems

View full text Add to dashboard Cite

Performance testing is a critical task to ensure an acceptable user experience with software systems, especially when there are high numbers of concurrent users. Selecting an appropriate test workload is a challenging and time-consuming process that relies heavily on the testers' expertise. Not only are workloads application-dependent, but also it is usually unclear how large a workload must be to expose any performance issues that exist in an application. Previous research has proposed to dynamically adapt the test workloads in real-time based on the application behavior. By reducing the need for the trial-and-error test cycles required when using static workloads, dynamic workload adaptation can reduce the effort and expertise needed to carry out performance testing. However, such approaches usually require testers to properly configure several parameters in order to be effective in identifying workloaddependent performance bugs, which may hinder their usability among practitioners. To address this issue, this paper examines the different criteria needed to conduct performance testing efficiently using dynamic workload adaptation. We present the results of comprehensively evaluating one such approach, providing insights into how to tune it properly in order to obtain better outcomes based on different scenarios. We also study the effects of varying its configuration and how this can affect the results obtained.

show abstract

How to Detect Performance Changes in Software History

Cited by 9 publications

References 21 publications

Performance mutation testing

Performance mutation testing

DYNAMOJM: A JMeter Tool for Performance Testing Using Dynamic Workload Adaptation

Towards an Efficient Performance Testing Through Dynamic Workload Adaptation

Contact Info

Product

Resources

About