Characterizing serverless platforms with serverlessbench

Abstract

Serverless computing promises auto-scalability and costefficiency (in "pay-as-you-go" manner) for high-productive software development. Because of its virtue, serverless computing has motivated increasingly new applications and services in the cloud. This, however, also presents new challenges including how to efficiently design high-performance serverless platforms and how to efficiently program on the platforms. This paper proposes ServerlessBench, an open-source benchmark suite for characterizing serverless platforms. It includes test cases exploring characteristic metrics of serverless computing, e.g., communication efficiency, startup latency, stateless overhead, and performance isolation. We have applied the benchmark suite to evaluate the most popular serverless computing platforms, including AWS Lambda, Open-Whisk, and Fn, and present new serverless implications from the study. For example, we show scenarios where decoupling an application into a composition of serverless functions can be beneficial in cost-saving and performance, and that the "stateless" property in serverless computing can hurt the execution performance of serverless functions. These implications form several design guidelines, which may help platform

Benchmarking, analysis, and optimization of serverless function snapshots

“…vHive adopts dockerized benchmarks from FunctionBench that provides a diverse set of Python benchmarks [32,33]. Server-lessBench contains a number of multi-function benchmarks, focusing on function composition patterns [65]. Researchers and practitioners release a range of systems that combine the FaaS programming model and autoscaling [8-10, 30, 36].…”

Ofc

“…For the former, we use 19 multimedia processing functions, available online (see our code repository in §1). For multi-stage functions, we study four applications: two data analytics applications as in [23] (a MapReduce-based "word count" application; Thousand Island Scanner (THIS) [33], as well as (in §7.2), a distributed video processing benchmark), a cloud-based Illegitimate Mobile App Detector (IMAD) application [45] 4 , and an image thumbnail generator pipeline (Image Processing) from the ServerlessBench suite [47].…”

SeMoDe – Simulation and Benchmarking Pipeline for Function as a Service

“…Similar to Maissen, they also found different CPU models executing the cloud function, which could also explain the scatter plot in Figure 5 in [55] (respectively Figure 4 in [53]), where two performance ranges for LINPACK were visible. The last included benchmarking paper in the SLR is another workbench, called ServerlessBench [56]. They included an experiment with an multi-threaded application compared to parallel function instances.…”

“…PaaS where single functionality in the scope of a cloud function is hidden within a larger application/microservice. Therefore, the multithreading aspect when optimizing performance and costs of cloud function is important to consider for FaaS experiments when using multi-core configurations [9,56]. This is different to long running applications in a PaaS area where multi-threaded implementations of pieces of functionality are suspended for code readability and maintenance which is also addressed by static code quality tools 21 .…”