Enhancing Search-based Testing with Testability Transformations for Existing APIs

ArcuriAndrea,; Galeotti, Juan Pablo

doi:10.1145/3477271

Cited by 25 publications

(12 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, there is no info to use an HTTP parameter called mc gross of type double. As such parameter is used directly without being transformed, testability transformations with taint analysis might be able to address this problem [28], but such techniques would need to be extended to support getParameterMap() and Map.get.…”

Section: Open Problemsmentioning

confidence: 99%

“…This fails, as a random string is extremely unlikely to represent a valid value from a restricted set. This might be handled by providing such info in the OpenAPI/Swagger schema (which supports defining enumerations on string fields), or also by handling valueOf() in enumeration by extending the techniques in [28] to support it.…”

Section: Open Problemsmentioning

confidence: 99%

“…But, it is extremely unlikely that a random string would represent a URI with a valid scheme component, and so uri.getScheme() returns null. However, this is a case that likely would be trivial to solve with taint analysis, e.g., by extending the techniques in [28] to support URI objects. This does not mean though that much higher coverage would be achieved, it depends on all the remaining code that is executed once that predicate is satisfied.…”

Section: Open Problemsmentioning

confidence: 99%

See 2 more Smart Citations

Open Problems in Fuzzing RESTful APIs: A Comparison of Tools

Zhang¹,

Arcuri²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

RESTful APIs are a type of web services that are widely used in industry. In the last few years, a lot of effort in the research community has been spent in designing novel techniques to automatically fuzz those APIs to find faults in them. Many real faults were automatically found in a large variety of RESTful APIs. However, usually the analyzed fuzzers treat the APIs as black-box, and no analysis of what is actually covered in these systems is done. Therefore, although these fuzzers are clearly useful for practitioners, we do not know what are their current limitations and actual effectiveness. Solving this is a necessary step to be able to design better, more efficient and effective techniques. To address this issue, in this paper we compare 6 state-of-the-art fuzzers on 10 RESTful APIs. We then analyzed the source code of which parts of these APIs the fuzzers fail to generate tests for. This analysis points to clear limitations of these current fuzzers, listing concrete challenges for the research community to follow up on.

show abstract

Section: Open Problemsmentioning

confidence: 99%

Section: Open Problemsmentioning

confidence: 99%

Section: Open Problemsmentioning

confidence: 99%

See 1 more Smart Citation

Open Problems in Fuzzing RESTful APIs: A Comparison of Tools

Zhang¹,

Arcuri²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Lin et al [30] proposed a test seed synthesis approach to create complex test inputs. Arcuri et al [4] integrated testability transformations into API tests. Braione et al [6] combined symbolic execution and SBST for programs with complex inputs; (3) readability of generated tests: Daka et al [11] proposed to assign names for tests by summarizing covered coverage goals.…”

Section: Related Workmentioning

confidence: 99%

Selectively Combining Multiple Coverage Goals in Search-Based Unit Test Generation

Zhou¹,

Zhou²,

Fang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Unit testing is a critical part of software development process, ensuring the correctness of basic programming units in a program (e.g., a method). Search-based software testing (SBST) is an automated approach to generating test cases. SBST generates test cases with genetic algorithms by specifying the coverage criterion (e.g., branch coverage). However, a good test suite must have different properties, which cannot be captured by using an individual coverage criterion. Therefore, the state-of-the-art approach combines multiple criteria to generate test cases. As combining multiple coverage criteria brings multiple objectives for optimization, it hurts the test suites' coverage for certain criteria compared with using the single criterion. To cope with this problem, we propose a novel approach named smart selection. Based on the coverage correlations among criteria and the coverage goals' subsumption relationships, smart selection selects a subset of coverage goals to reduce the number of optimization objectives and avoid missing any properties of all criteria. We conduct experiments to evaluate smart selection on 400 Java classes with three state-of-the-art genetic algorithms. On average, smart selection outperforms combining all goals on 65.1% of the classes having significant differences between the two approaches. CCS CONCEPTS• Software and its engineering → Search-based software engineering; Software testing and debugging.

show abstract

“…But, most of these open-source APIs are web services that work in isolation, and not part of a microservice architecture [2]. As part of industry-driven research [27], EvoMaster has been evaluated on industrial APIs as well (e.g., in [15]). But those were part of small systems.…”

Section: Introductionmentioning

confidence: 99%

Fuzzing Microservices In Industry: Experience of Applying EvoMaster at Meituan

Zhang¹,

Arcuri²,

Li³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

With several microservice architectures comprising of thousands of web services in total, used to serve 630 million customers, companies like Meituan face several challenges in the verification and validation of their software. The use of automated techniques, especially advanced AI-based ones, could bring significant benefits here. EvoMaster is an open-source test case generation tool for web services, that exploits the latest advances in the field of Search-Based Software Testing research. This paper reports on our experience of integrating the EvoMaster tool in the testing processes at Meituan. Experiments were carried out to evaluate its performance in detail on two industrial web services which are parts of a large e-commerce microservices. A questionnaire and interviews were carried out with the engineers and managers at Meituan, to better understand the applicability and usability of fuzzing tools like EvoMaster in real industrial settings. On the one hand, the results of these analyses clearly show that existing tools like EvoMaster are already of benefits for practitioners in industry, e.g., EvoMaster detected 21 real faults and achieved an average of 71.3% coverage for code defining endpoints and 51.7% coverage for code implementing business logic, on two industrial APIs. On the other hand, there are still many critical challenges that the research community has to investigate.

show abstract

Enhancing Search-based Testing with Testability Transformations for Existing APIs

Cited by 25 publications

References 56 publications

Open Problems in Fuzzing RESTful APIs: A Comparison of Tools

Open Problems in Fuzzing RESTful APIs: A Comparison of Tools

Selectively Combining Multiple Coverage Goals in Search-Based Unit Test Generation

Fuzzing Microservices In Industry: Experience of Applying EvoMaster at Meituan

Contact Info

Product

Resources

About