Automated Performance Testing Based on Active Deep Learning

Sedaghatbaf, Ali; Moghadam, Mahshid Helali; Saadatmand, Mehrdad

doi:10.1109/ast52587.2021.00010

Cited by 10 publications

(5 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Performance can be measured, which offers feedback for subsequent rounds of generation. Thus, the majority of approaches are based on iterative processes, including reinforcement [22,[85][86][87], rule [88], and adversarial learning [89].…”

Section: Performance Test Generationmentioning

confidence: 99%

“…Sedaghatbaf et al generate input violating performance requirements using two competing neural networks [89]. The generator produces input, and the discriminator classifies whether input violates requirements.…”

Section: Performance Test Generationmentioning

confidence: 99%

“…The most common, by far, is the F-Droid Android benchmark (e.g., previous studies [60,61]). Others made use of examples commonly used in research such as the Defects4J (e.g., previous studies [79,111]) or the RUBiS web app example (e.g., Sedaghatbaf et al [89]). However, the majority of studies do not use benchmarks or open-source evaluation targets.…”

Section: Rq6 (Challenges)mentioning

confidence: 99%

See 2 more Smart Citations

The integration of machine learning into automated test generation: A systematic mapping study

Fontes

Gay

2023

Software Testing Verif & Rel

View full text Add to dashboard Cite

Machine learning (ML) may enable effective automated test generation. We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges in this intersection by performing. We perform a systematic mapping study on a sample of 124 publications. ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property‐based, and expected output oracles. Supervised learning—often based on neural networks—and reinforcement learning—often based on Q‐learning—are common, and some publications also employ unsupervised or semi‐supervised learning. (Semi‐/Un‐)Supervised approaches are evaluated using both traditional testing metrics and ML‐related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. The work‐to‐date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed—and how they are applied—benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.

show abstract

Section: Performance Test Generationmentioning

confidence: 99%

Section: Performance Test Generationmentioning

confidence: 99%

Section: Rq6 (Challenges)mentioning

confidence: 99%

See 1 more Smart Citation

The integration of machine learning into automated test generation: A systematic mapping study

Fontes

Gay

2023

Software Testing Verif & Rel

View full text Add to dashboard Cite

show abstract

“…rule [69], and adversarial learning [68]. Rather than generating input, [15,67] apply Q-Learning to control the execution environment.…”

Section: Performance Test Generationmentioning

confidence: 99%

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

Fontes¹,

Gay²

2022

Preprint

View full text Add to dashboard Cite

Context: Machine learning (ML) may enable effective automated test generation.Objectives: We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges.Methods: We perform a systematic literature review on a sample of 97 publications.Results: ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property-based, and expected output oracles. Supervised learning-often based on neural networks-and reinforcement learning-often based on Q-learning-are common, and some publications also employ unsupervised or semi-supervised learning. (Semi-/Un-)Supervised approaches are evaluated using both traditional testing metrics and ML-related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. Conclusion:Work-to-date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed-and how they are applied-benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.

show abstract

“…Deep neural networks (DNNs) [38] have been increasingly adopted in many fields, including computer vision [5], natural language processing [19], software engineering [13,18,32,39,45,48], etc. However, one of the crucial factors hindering DNNs from further serving applications with social impact is the unintended individual discrimination [44,47,55].…”

Section: Introductionmentioning

confidence: 99%

NeuronFair: Interpretable White-Box Fairness Testing through Biased Neuron Identification

Zheng¹,

Chen²,

Du³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep neural networks (DNNs) have demonstrated their outperformance in various domains. However, it raises a social concern whether DNNs can produce reliable and fair decisions especially when they are applied to sensitive domains involving valuable resource allocation, such as education, loan, and employment. It is crucial to conduct fairness testing before DNNs are reliably deployed to such sensitive domains, i.e., generating as many instances as possible to uncover fairness violations. However, the existing testing methods are still limited from three aspects: interpretability, performance, and generalizability. To overcome the challenges, we propose NeuronFair, a new DNN fairness testing framework that differs from previous work in several key aspects: (1) interpretable -it quantitatively interprets DNNs' fairness violations for the biased decision; (2) effective -it uses the interpretation results to guide the generation of more diverse instances in less time; (3) generic -it can handle both structured and unstructured data. Extensive evaluations across 7 datasets and the corresponding DNNs demonstrate NeuronFair's superior performance. For instance, on structured datasets, it generates much more instances (∼ ×5.84) and saves more time (with an average speedup of 534.56%) compared with the state-of-the-art methods. Besides, the instances of NeuronFair can also be leveraged to improve the fairness of the biased DNNs, which helps build more fair and trustworthy deep learning systems. The code of NeuronFair is open-sourced at https:// github.com/ haibinzheng/ NeuronFair. CCS CONCEPTS• Computing methodologies → Artificial intelligence; • Software and its engineering → Software reliability.

show abstract

Automated Performance Testing Based on Active Deep Learning

Cited by 10 publications

References 22 publications

The integration of machine learning into automated test generation: A systematic mapping study

The integration of machine learning into automated test generation: A systematic mapping study

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

NeuronFair: Interpretable White-Box Fairness Testing through Biased Neuron Identification

Contact Info

Product

Resources

About