Using regression learners to predict performance problems on software updates

Gartziandia, Aitor; Arrieta, Aitor; Agirre, Aitor; Sagardui, Goiuria; Arratibel, Maite

doi:10.1145/3412841.3441894

Cited by 7 publications

(7 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The verdict from the ANN oracle is calculated as the Mean Squared Error (MSE) between the real and predicted outputs, which means that the oracle can calculate quantitative verdicts [56]. We also proposed the application of machine learning algorithms to alleviate the test oracle problem in the domain of elevation, both for functional [3] as well as non-functional faults [25]. Other domain-specific approaches based on machine learning have also been proposed, such as an unsupervised approach for autonomous vehicles testing [58].…”

Section: B Testing Cpssmentioning

confidence: 99%

Performance-Driven Metamorphic Testing of Cyber-Physical Systems

et al. 2023

View full text Add to dashboard Cite

Cyber-Physical Systems (CPSs) are a new generation of systems which integrate software with physical processes. The increasing complexity of these systems, combined with the uncertainty in their interactions with the physical world, makes the definition of effective test oracles especially challenging, facing the well known test oracle problem. Metamorphic testing has shown great potential to alleviate the test oracle problem by exploiting the relations among the inputs and outputs of different executions of the system, so-called metamorphic relations (MRs). In this article, we propose a MR pattern called Performance Variation (PV) for the identification of performance-driven MRs, and we show its applicability in two CPSs from different domains: automated navigation systems and elevator control systems. For the evaluation, we assessed the effectiveness of this approach for detecting failures in an open source simulation-based autonomous navigation system, as well as in an industrial case study from the elevation domain. We derive concrete MRs based on the PV pattern for both case studies and we evaluate their effectiveness with seeded faults. Results show that the approach is effective at detecting over 88% of the seeded faults, while keeping the ratio of false positives at 4% or lower.

show abstract

Section: B Testing Cpssmentioning

confidence: 99%

Performance-Driven Metamorphic Testing of Cyber-Physical Systems

et al. 2023

View full text Add to dashboard Cite

show abstract

“…29 However, that metamorphic testing approach was mainly designed for short-scenario tests, whereas the technique shown in this paper is designed for long-scenario tests. Our previous work used ML algorithms to detect performance bugs on software updates 77 ; nevertheless, the evaluation was not as realistic as in this case, because the execution of the algorithms was performed in a local PC, whereas in this case, we used the real microprocessor used by Orona in their elevators. In fact, we noticed that the results were significantly different in this case.…”

Section: Testing Of Systems Of Elevatorsmentioning

confidence: 99%

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

Gartziandia

Arrieta

Ayerdi

et al. 2022

J Software Evolu Process

View full text Add to dashboard Cite

The software of systems of elevators needs constant maintenance to deal with new functionality, bug fixes, or legislation changes. To automatically validate the software of these systems, a typical approach in industry is to use regression oracles, which execute test inputs both in the software version under test and in a previous software version. However, these practices require a long test execution time and cannot be re‐used at different test phases. To deal with these issues, we propose Dispatching AlgoRIthm Oracle (DARIO), a test oracle that relies on regression machine‐learning algorithms to detect both functional and non‐functional problems of the system. The machine‐learning algorithms of this oracle are trained by using data from previously tested versions to predict reference functional and non‐functional performance values of the new versions. An empirical evaluation with an industrial case study demonstrates the feasibility of using our approach. A total of five regression learning algorithms were validated by using mutation testing techniques. For the context of functional bugs, the accuracy when predicting verdicts by DARIO ranged between 95% and 98%, across the different scenarios proposed. For the context of non‐functional bugs, were competitive too, having an accuracy when predicting verdicts by DARIO ranged between 83% and 87%.

show abstract

“…Recently, however, [84,86,88,95] generate oracles for functions with unconstrainede.g., integer-output.…”

Section: Test Oracle Generationmentioning

confidence: 99%

“…[84, 86] compared regression trees, SVM, an ensemble model, a Regression Gaussian Process (RGP), and a stepwise regression. [84] found regression tree to be the best, while [86] found regression tree, ensemble, and RGP valid.…”

Section: Test Oracle Generationmentioning

confidence: 99%

See 1 more Smart Citation

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

Fontes¹,

Gay²

2022

Preprint

View full text Add to dashboard Cite

Context: Machine learning (ML) may enable effective automated test generation.Objectives: We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges.Methods: We perform a systematic literature review on a sample of 97 publications.Results: ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property-based, and expected output oracles. Supervised learning-often based on neural networks-and reinforcement learning-often based on Q-learning-are common, and some publications also employ unsupervised or semi-supervised learning. (Semi-/Un-)Supervised approaches are evaluated using both traditional testing metrics and ML-related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. Conclusion:Work-to-date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed-and how they are applied-benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.

show abstract

Using regression learners to predict performance problems on software updates

Cited by 7 publications

References 27 publications

Performance-Driven Metamorphic Testing of Cyber-Physical Systems

Performance-Driven Metamorphic Testing of Cyber-Physical Systems

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

Contact Info

Product

Resources

About