Microservices: A Performance Tester's Dream or Nightmare?

Eismann, Simon; Bezemer, Cor‐Paul; Shang, Weiyi; Okanović, Dušan; Hoorn, André van

doi:10.1145/3358960.3379124

Cited by 26 publications

(14 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For this, we used the keyword and file type search on GitHub. Additionally, we searched the API repository from APIs.guru 11 , which provides a substantial number of OpenAPI files.…”

Section: Methodsmentioning

confidence: 99%

“…In systems based on service orientation [22], however, many source code metrics lose their importance due to the increased level of abstraction [4]. For microservices as a lightweight and fine-grained service-oriented variant [20], factors like the large number of small services, their decentralized nature, or high degree of technological heterogeneity may pose difficulties for metric collection and the applicability of existing metrics, which has also been reported in the area of performance testing [11]. Several researchers have therefore focused on adapting existing metrics and defining new metrics for service orientation (see e.g.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Collecting Service-Based Maintainability Metrics from RESTful API Descriptions: Static Analysis and Threshold Derivation

Bogner,

Wagner,

Zimmermann

2020

Preprint

View full text Add to dashboard Cite

While many maintainability metrics have been explicitly designed for service-based systems, tool-supported approaches to automatically collect these metrics are lacking. Especially in the context of microservices, decentralization and technological heterogeneity may pose challenges for static analysis. We therefore propose the modular and extensible RAMA approach (RESTful API Metric Analyzer) to calculate such metrics from machine-readable interface descriptions of RESTful services. We also provide prototypical tool support, the RAMA CLI, which currently parses the formats OpenAPI, RAML, and WADL and calculates 10 structural service-based metrics proposed in scientific literature. To make RAMA measurement results more actionable, we additionally designed a repeatable benchmark for quartile-based threshold ranges (green, yellow, orange, red). In an exemplary run, we derived thresholds for all RAMA CLI metrics from the interface descriptions of 1,737 publicly available RESTful APIs. Researchers and practitioners can use RAMA to evaluate the maintainability of RESTful services or to support the empirical evaluation of new service interface metrics.

show abstract

“…For this, we used the keyword and file type search on GitHub. Additionally, we searched the API repository from APIs.guru 11 , which provides a substantial number of OpenAPI files.…”

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Collecting Service-Based Maintainability Metrics from RESTful API Descriptions: Static Analysis and Threshold Derivation

Bogner,

Wagner,

Zimmermann

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…They used the developed model to perform what-if analysis and capacity planning for largescale microservices. Eismann et al [37] demonstrated the benefits and challenges that arise in the performance testing of microservices and how to manage the unique complications that arise while doing so. Kaviani et al [38] discusses the effectiveness of several key components of Knative and its contribution to opensource serverless computing platforms.…”

Section: Related Workmentioning

confidence: 99%

Performance Modeling of Serverless Computing Platforms

Mahmoudi

Khazaei

2022

IEEE Trans. Cloud Comput.

View full text Add to dashboard Cite

Developing accurate and extendable performance models for serverless platforms, aka Function-as-a-Service (FaaS) platforms, is a very challenging task. Also, implementation and experimentation on real serverless platforms is both costly and time-consuming. However, at the moment, there is no comprehensive simulation tool or framework to be used instead of the real platform. As a result, in this paper, we fill this gap by proposing a simulation platform, called SimFaaS, which assists serverless application developers to develop optimized Function-as-a-Service applications in terms of cost and performance. On the other hand, SimFaaS can be leveraged by FaaS providers to tailor their platforms to be workload-aware so that they can increase profit and quality of service at the same time. Also, serverless platform providers can evaluate new designs, implementations, and deployments on SimFaaS in a timely and cost-efficient manner. SimFaaS is open-source, well-documented, and publicly available, making it easily usable and extendable to incorporate more use case scenarios in the future. Besides, it provides performance engineers with a set of tools that can calculate several characteristics of serverless platform internal states, which is otherwise hard (mostly impossible) to extract from real platforms. In previous studies, temporal and steady-state performance models for serverless computing platforms have been developed. However, those models are limited to Markovian processes. We designed SimFaaS as a tool that can help overcome such limitations for performance and cost prediction in serverless computing. We show how SimFaaS facilitates the prediction of essential performance metrics such as average response time, probability of cold start, and the average number of instances reflecting the infrastructure cost incurred by the serverless computing provider. We evaluate the accuracy and applicability of SimFaaS by comparing the prediction results with real-world traces from Amazon AWS Lambda.

show abstract

“…We address these challenges by (i) automatically extracting and evolving performance tests using operational monitoring data and API information [14,17], (ii) the generation and selection of tailored tests based on current test concerns [12,13], as well as (iii) exploiting recommending test strategies suitable for testing in unreliable infrastructures [2,9].…”

Section: Continuous Testingmentioning

confidence: 99%

Performance Engineering for Microservices and Serverless Applications: The RADON Approach

Gias

Hoorn

Zhu

et al. 2020

Companion of the ACM/SPEC International Conference on Performance Engineering

Self Cite

View full text Add to dashboard Cite

Microservices and serverless are becoming integral parts of modern cloud-based applications. Tailored performance engineering is needed for assuring that the applications meet their requirements for quality attributes such as timeliness, resource efficiency, and elasticity. A novel DevOps-based framework for developing microservices and serverless applications is being developed in the RADON project. RADON contributes to performance engineering by including novel approaches for modeling, deployment optimization, testing, and runtime management. This paper summarizes the contents of our tutorial presented at the 11th ACM/SPEC International Conference on Performance Engineering (ICPE).

show abstract

Microservices: A Performance Tester's Dream or Nightmare?

Cited by 26 publications

References 33 publications

Collecting Service-Based Maintainability Metrics from RESTful API Descriptions: Static Analysis and Threshold Derivation

Collecting Service-Based Maintainability Metrics from RESTful API Descriptions: Static Analysis and Threshold Derivation

Performance Modeling of Serverless Computing Platforms

Performance Engineering for Microservices and Serverless Applications: The RADON Approach

Contact Info

Product

Resources

About