Automated black‐box testing of nominal and error scenarios in RESTful APIs

Corradini, Davide; Zampieri, Amedeo; Pasqua, Michele; Viglianisi, Emanuele; Dallago, Michael; Ceccato, Mariano

doi:10.1002/stvr.1808

Cited by 30 publications

(5 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Testing REST APIs. Approaches for automated black-box testing of REST APIs are proposed in the literature, including RESTest [32,33], ARTE [7] -an extension to RESTest, RESTler [12,22], Semanthesis [24], RESTCT [42], RestTestGen [14,15], EvoMaster [9,10], RapiTest [19], and QuickREST [26]. Our study utilizes RESTest, as it is a black-box REST APIs testing approach, which supports realistic test data generation [7] and has shown good results in testing online applications [33].…”

Section: Related Workmentioning

confidence: 99%

Testing Real-World Healthcare IoT Application: Experiences and Lessons Learned

Sartaj,

Ali,

Yue

et al. 2023

Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Enginee

View full text Add to dashboard Cite

Healthcare Internet of Things (IoT) applications require rigorous testing to ensure their dependability. Such applications are typically integrated with various third-party healthcare applications and medical devices through REST APIs. This integrated network of healthcare IoT applications leads to REST APIs with complicated and interdependent structures, thus creating a major challenge for automated system-level testing. We report an industrial evaluation of a state-of-the-art REST APIs testing approach (RESTest) on a real-world healthcare IoT application. We analyze the effectiveness of RESTest's testing strategies regarding REST APIs failures, faults in the application, and REST API coverage, by experimenting with six REST APIs of 41 API endpoints of the healthcare IoT application. Results show that several failures are discovered in different REST APIs with ≈56% coverage using RESTest. Moreover, nine potential faults are identified. Using the evidence collected from the experiments, we provide our experiences and lessons learned. CCS CONCEPTS• Software and its engineering → Software testing and debugging; Software verification and validation.

show abstract

Section: Related Workmentioning

confidence: 99%

Testing Real-World Healthcare IoT Application: Experiences and Lessons Learned

Sartaj,

Ali,

Yue

et al. 2023

Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Enginee

View full text Add to dashboard Cite

show abstract

“…RestTestGen [50,35] is another black-box fuzzer for RESTful APIs. It is written in Java, but the original versions used in [50,35] are not opensource.…”

Section: Introductionmentioning

confidence: 99%

“…RestTestGen [50,35] is another black-box fuzzer for RESTful APIs. It is written in Java, but the original versions used in [50,35] are not opensource. Since late 2021, a new rewrite of the tool as open-source seems underway.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Open Problems in Fuzzing RESTful APIs: A Comparison of Tools

Zhang¹,

Arcuri²

2022

Preprint

View full text Add to dashboard Cite

RESTful APIs are a type of web services that are widely used in industry. In the last few years, a lot of effort in the research community has been spent in designing novel techniques to automatically fuzz those APIs to find faults in them. Many real faults were automatically found in a large variety of RESTful APIs. However, usually the analyzed fuzzers treat the APIs as black-box, and no analysis of what is actually covered in these systems is done. Therefore, although these fuzzers are clearly useful for practitioners, we do not know what are their current limitations and actual effectiveness. Solving this is a necessary step to be able to design better, more efficient and effective techniques. To address this issue, in this paper we compare 6 state-of-the-art fuzzers on 10 RESTful APIs. We then analyzed the source code of which parts of these APIs the fuzzers fail to generate tests for. This analysis points to clear limitations of these current fuzzers, listing concrete challenges for the research community to follow up on.

show abstract

“…After doing so, ARTE is able to query semantic knowledge databases during the process of input generation. By following the approach just described, ARTECorradini et al (2022) presents RestTestGen, a new approach for automatically generating tests for REST APIs from their corresponding OpenAPI specifications. RestTest-Gen is able to generate both nominal and error test cases.…”

mentioning

confidence: 99%

OAS DB: a shared infrastructure to support OpenAPI research

Stoll¹

View full text Add to dashboard Cite

It is common knowledge the great success achieved by the Web in the last decades. Together with the rise of Web systems in general, it came the increase of the number of Web APIs. There are many specifications used to describe an Web API. One of the most popular ones is OpenAPI. This specification allows one to describe all the resources that can be accessed and manipulated through a REST Web API. An OpenAPI specification can be used to perform different kinds of analysis and verification of the service implementing the described API. A common challenge faced by researchers, however, is the lack of shared validation infrastructure or a standard benchmark. The main contribution of our research is a software artifact -called OAS DB (OpenAPI Specifications Database) -that aims to provide researchers and industry practitioners with a complete solution to streamline the validation of new OpenAPI related techniques and tools. OAS DB is able to generate complete OpenAPI specifications and their corresponding mock implementations. It is also both capable of injecting faults and anti-patterns in these generated specifications/mock implementations and of indicating -through machine-readable files -which issues and anti-patterns are present in the generated assets. We use OAS DB to assess tools relying on both static and dynamic techniques to detect faults and anti-patterns in OpenAPI specifications. Our results indicate that these tools fail to detect relevant faults and anti-patterns in the synthetic APIs generated by OAS DB, indicating that there is room to improve these tools and the ways in which they are applying static and dynamic analysis techniques. The present work also has as contributions: a) a proof of concept REST API anti-pattern detector (which we call Oasis) and b) the description of a novel REST anti-pattern not described in the literature so far.

show abstract

Automated black‐box testing of nominal and error scenarios in RESTful APIs

Cited by 30 publications

References 29 publications

Testing Real-World Healthcare IoT Application: Experiences and Lessons Learned

Testing Real-World Healthcare IoT Application: Experiences and Lessons Learned

Open Problems in Fuzzing RESTful APIs: A Comparison of Tools

OAS DB: a shared infrastructure to support OpenAPI research

Contact Info

Product

Resources

About