Adaptive Hypermutation for Search-Based System Test Generation: A Study on REST APIs with EvoMaster

Abstract

REST web services are widely popular in industry, and search techniques have been successfully used to automatically generate system-level test cases for those systems. In this article, we propose a novel mutation operator which is designed specifically for test generation at system-level, with a particular focus on REST APIs. In REST API testing, and often in system testing in general, an individual can have a long and complex chromosome. Furthermore, there are two specific issues: (1) fitness evaluation in system testing is highly costly compared with the number of objectives (e.g., testing targets) to optimize for; and (2) a large part of the genotype might have no impact on the phenotype of the individuals (e.g., input data that has no impact on the execution flow in the tested program). Due to these issues, it might be not suitable to apply a typical low mutation rate like 1/ n (where n is the number of genes in an individual), which would lead to mutating only one gene on average. Therefore, in this article, we propose an adaptive weight-based hypermutation, which is aware of the different characteristics of the mutated genes. We developed adaptive strategies that enable the selection and mutation of genes adaptively based on their fitness impact and mutation history throughout the search. To assess our novel proposed mutation operator, we implemented it in the EvoMaster tool, integrated in the MIO algorithm, and further conducted an empirical study with three artificial REST APIs and four real-world REST APIs. Results show that our novel mutation operator demonstrates noticeable improvements over the default MIO. It provides a significant improvement in performance for six out of the seven case studies, where the relative improvement is up to +12.09% for target coverage, +12.69% for line coverage, and +32.51% for branch coverage.

Building an open-source system test generation tool: lessons learned and empirical analyses with EvoMaster

“…Being able to add data directly into the database might not be so important if the testing tool is able to effectively use the endpoints of the API itself to create the needed data. As EvoMastEr has been improved since we carried out the analyses in Arcuri and Galeotti (2020a), the introduced novel techniques like resource handling (Zhang et al, 2021) and adaptive hypermutation (Zhang & Arcuri, 2021a) might have been enough for this specific SUT.…”

“…EvoMastEr uses the evolutionary algorithm MIO (Arcuri, 2018c) enhanced with adaptive hyper-mutation (Zhang & Arcuri, 2021a) to evolve test cases. As it is a randomized algorithm, when running a search there is no guarantee that an optimal solution is found.…”

Fuzzing Microservices In Industry: Experience of Applying EvoMaster at Meituan

“…Throughout the years, since its inception in 2016, EvoMaster has been extended with various novel techniques [14,16,55,51,21] to serve a more comprehensive testing in the context of white-box testing for enterprise systems.…”

“…In this context, a test (also referred as an individual) with additional SQL commands could result in a long and complex chromosome to evolve. To have an effective mutation for handling such individuals, an adaptive weight-based hypermutation [51] was developed that comprises a set of novel strategies to adaptively manipulate the number of genes to mutate, select genes to mutate, and guide how to mutate the values in these genes.…”

“…There is no guarantee that the implementation of EvoMaster is bug-free. However, EvoMaster has been used to perform many experiments using open-source case studies [12,14,16,51], and it is also carefully tested, i.e., currently with 652 test suites (including unit tests and end-to-end tests), which covers 60.65% lines of its code base. In addition, EvoMaster is an open-source project which can be reviewed and examined by anyone who is interested in it.…”