RiverFuzzRL - an open-source tool to experiment with reinforcement learning for fuzzing

Păduraru, Ciprian; Paduraru, Miruna; Ştefănescu, Alin

doi:10.1109/icst49551.2021.00055

Cited by 8 publications

(2 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Esnaashari and Damia use reinforcement learning to manipulate tests within the population generated by the Genetic Algorithm by modifying their input [27]. Paduraru et al similarly use reinforcement learning to improve the effectiveness of a random testing tool by taking generated input and modifying it to raise its coverage or execution path length [49]. Zhong et al bias input selection in a fuzzing tool for autonomous driving simulators by learning which seeds for the random number generator are more likely to lead to traffic violations in the simulation [46].…”

Section: System Test Generationmentioning

confidence: 99%

The integration of machine learning into automated test generation: A systematic mapping study

Fontes

Gay

2023

Software Testing Verif & Rel

View full text Add to dashboard Cite

Machine learning (ML) may enable effective automated test generation. We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges in this intersection by performing. We perform a systematic mapping study on a sample of 124 publications. ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property‐based, and expected output oracles. Supervised learning—often based on neural networks—and reinforcement learning—often based on Q‐learning—are common, and some publications also employ unsupervised or semi‐supervised learning. (Semi‐/Un‐)Supervised approaches are evaluated using both traditional testing metrics and ML‐related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. The work‐to‐date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed—and how they are applied—benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.

show abstract

Section: System Test Generationmentioning

confidence: 99%

The integration of machine learning into automated test generation: A systematic mapping study

Fontes

Gay

2023

Software Testing Verif & Rel

View full text Add to dashboard Cite

show abstract

“…[21] use RL to manipulate tests within the GA by modifying input. [40] similarly use RL to improve the effectiveness of a fuzzing tool. RL modifies input selected by the fuzzing algorithm to improve either code coverage or longest execution path length.…”

Section: Examining Specific Practicesmentioning

confidence: 99%

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

Fontes¹,

Gay²

2022

Preprint

View full text Add to dashboard Cite

Context: Machine learning (ML) may enable effective automated test generation.Objectives: We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges.Methods: We perform a systematic literature review on a sample of 97 publications.Results: ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property-based, and expected output oracles. Supervised learning-often based on neural networks-and reinforcement learning-often based on Q-learning-are common, and some publications also employ unsupervised or semi-supervised learning. (Semi-/Un-)Supervised approaches are evaluated using both traditional testing metrics and ML-related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. Conclusion:Work-to-date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed-and how they are applied-benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.

show abstract

Building an open-source system test generation tool: lessons learned and empirical analyses with EvoMaster

et al. 2023

View full text Add to dashboard Cite

Research in software testing often involves the development of software prototypes. Like any piece of software, there are challenges in the development, use and verification of such tools. However, some challenges are rather specific to this problem domain. For example, often these tools are developed by PhD students straight out of bachelor/master degrees, possibly lacking any industrial experience in software development. Prototype tools are used to carry out empirical studies, possibly studying different parameters of novel designed algorithms. Software scaffolding is needed to run large sets of experiments efficiently. Furthermore, when using AI-based techniques like evolutionary algorithms, care needs to be taken to deal with their randomness, which further complicates their verification. The aforementioned represent some of the challenges we have identified for this domain. In this paper, we report on our experience in building the open-source EvoMaster tool, which aims at system-level test case generation for enterprise applications. Many of the challenges we faced would be common to any researcher needing to build software testing tool prototypes. Therefore, one goal is that our shared experience here will boost the research community, by providing concrete solutions to many development challenges in the building of such kind of research prototypes. Ultimately, this will lead to increase the impact of scientific research on industrial practice.

show abstract

RiverFuzzRL - an open-source tool to experiment with reinforcement learning for fuzzing

Cited by 8 publications

References 13 publications

The integration of machine learning into automated test generation: A systematic mapping study

The integration of machine learning into automated test generation: A systematic mapping study

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

Building an open-source system test generation tool: lessons learned and empirical analyses with EvoMaster

Contact Info

Product

Resources

About