Beyond Microbenchmarks: The SPEC-RG Vision for a Comprehensive Serverless Benchmark

Eyk, Erwin van; Scheuner, Joel; Eismann, Simon; Abad, Cristina L.; Iosup, Alexandru

doi:10.1145/3375555.3384381

Cited by 14 publications

(5 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In serverless-specific field, Microsoft researchers characterize the serverless workloads on Azure Functions [46]. FunctionBench [37] and SPEC-RG Vision [51] propose serverless benchmark designs to reflect various aspects of serverless computing. Previous works also evaluate and compare how serverless applications behave on different platforms [23,30,39].…”

Section: Related Workmentioning

confidence: 99%

Characterizing serverless platforms with serverlessbench

et al. 2020

Proceedings of the 11th ACM Symposium on Cloud Computing

136

View full text Add to dashboard Cite

Serverless computing promises auto-scalability and costefficiency (in "pay-as-you-go" manner) for high-productive software development. Because of its virtue, serverless computing has motivated increasingly new applications and services in the cloud. This, however, also presents new challenges including how to efficiently design high-performance serverless platforms and how to efficiently program on the platforms. This paper proposes ServerlessBench, an open-source benchmark suite for characterizing serverless platforms. It includes test cases exploring characteristic metrics of serverless computing, e.g., communication efficiency, startup latency, stateless overhead, and performance isolation. We have applied the benchmark suite to evaluate the most popular serverless computing platforms, including AWS Lambda, Open-Whisk, and Fn, and present new serverless implications from the study. For example, we show scenarios where decoupling an application into a composition of serverless functions can be beneficial in cost-saving and performance, and that the "stateless" property in serverless computing can hurt the execution performance of serverless functions. These implications form several design guidelines, which may help platform

show abstract

Section: Related Workmentioning

confidence: 99%

Characterizing serverless platforms with serverlessbench

et al. 2020

Proceedings of the 11th ACM Symposium on Cloud Computing

136

View full text Add to dashboard Cite

show abstract

“…Although including a cost model in cloud benchmarking studies is considered good scientific practice [37], in existing benchmarking studies on FaaS [24,[38][39][40][41] and DSP [4,42,43], cost evaluations can mainly be found for cloud functions, where the pay-perexecution pricing model has presented a significant paradigm shift.…”

Section: Related Workmentioning

confidence: 99%

Streaming vs. Functions: A Cost Perspective on Cloud Event Processing

Pfandzelter¹,

Henning²,

Schirmer³

et al. 2022

Preprint

View full text Add to dashboard Cite

In cloud event processing, data generated at the edge is processed in real-time by cloud resources. Both distributed stream processing (DSP) and Functionas-a-Service (FaaS) have been proposed to implement such event processing applications. FaaS emphasizes fast development and easy operation, while DSP emphasizes efficient handling of large data volumes. Despite their architectural differences, both can be used to model and implement loosely-coupled job graphs.In this paper, we consider the selection of FaaS and DSP from a cost perspective. We implement stateless and stateful workflows from the Theodolite benchmarking suite using cloud FaaS and DSP. In an extensive evaluation, we show how application type, cloud service provider, and runtime environment can influence the cost of application deployments and derive decision guidelines for cloud engineers.

show abstract

“…Although PanOpticon has similar goals as BeFaaS, it neither supports detailed drill-down analysis nor federated multi-provider setups. Van Eyk et al [31] develop a high-level architecture and state requirements for serverless benchmarking. Unfortunately their project still appears to be in a vision state.…”

Section: Related Workmentioning

confidence: 99%

BeFaaS: An Application-Centric Benchmarking Framework for FaaS Platforms

Grambow

Pfandzelter

Burchard

et al. 2021

2021 IEEE International Conference on Cloud Engineering (IC2E)

View full text Add to dashboard Cite

Due to the popularity of the FaaS programming model, there is now a wide variety of commercial and open-source FaaS systems. Hence, for comparison of different FaaS systems and their configuration options, FaaS application developers rely on FaaS benchmarking frameworks. Existing frameworks, however, tend to evaluate only single isolated aspects, a more holistic application-centric benchmarking framework is still missing.In previous work, we proposed BeFaaS, an extensible application-centric benchmarking framework for FaaS environments that focuses on the evaluation of FaaS platforms through realistic and typical examples of FaaS applications. In this extended paper, we (i) enhance our benchmarking framework with additional features for distributed FaaS setups, (ii) design application benchmarks reflecting typical FaaS use cases, and (iii) use them to run extensive experiments with commercial cloud FaaS platforms (AWS Lambda, Azure Functions, Google Cloud Functions) and the tinyFaaS edge serverless platform. BeFaaS now includes four FaaS application-centric benchmarks, is extensible for additional workload profiles and platforms, and supports federated benchmark runs in which the benchmark application is distributed over multiple FaaS systems while collecting fine-grained measurement results for drill-down analysis.Our experiment results show that (i) network transmission is a major contributor to response latency for function chains, (ii) this effect is exacer- * This work extends [1]. bated in hybrid edge-cloud deployments, (iii) the trigger delay between a published event and the start of the triggered function ranges from about 100ms for AWS Lambda to 800ms for Google Cloud Functions, and (iv) Azure Functions shows the best cold start behavior for our workloads.

show abstract

Beyond Microbenchmarks: The SPEC-RG Vision for a Comprehensive Serverless Benchmark

Cited by 14 publications

References 10 publications

Characterizing serverless platforms with serverlessbench

Characterizing serverless platforms with serverlessbench

Streaming vs. Functions: A Cost Perspective on Cloud Event Processing

BeFaaS: An Application-Centric Benchmarking Framework for FaaS Platforms

Contact Info

Product

Resources

About