Benchmarking Pocket-Scale Databases

Nuessle, Carl; Kennedy, Oliver; Ziarek, Lukasz

doi:10.1007/978-3-030-55024-0_7

Cited by 2 publications

(1 citation statement)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our experiments, we execute the well-established YCSB and TPC-C benchmarks, which are supported by the OLTPBench harness out-of-the-box. However, further special-purpose benchmarks have been proposed specifically for measuring tail latencies, as they arise in embedded databases on mobile devices [38]. This contribution is related to our work insofar as the authors also unveil sources of measurement error, however, errors that arise when measuring the performance of embedded databases at low throughput.…”

Section: Related Workmentioning

confidence: 91%

Tell-Tale Tail Latencies: Pitfalls and Perils in Database Benchmarking

Fruth¹,

Scherzinger²,

Mauerer³

et al. 2021

Preprint

View full text Add to dashboard Cite

The performance of database systems is usually characterised by their average-case (i.e., throughput) behaviour in standardised or defacto standard benchmarks like TPC-X or YCSB. While tails of the latency (i.e., response time) distribution receive considerably less attention, they have been identified as a threat to the overall system performance: In large-scale systems, even a fraction of requests delayed can build up into delays perceivable by end users. To eradicate large tail latencies from database systems, the ability to faithfully record them, and likewise pinpoint them to the root causes, is imminently required. In this paper, we address the challenge of measuring tail latencies using standard benchmarks, and identify subtle perils and pitfalls. In particular, we demonstrate how Java-based benchmarking approaches can substantially distort tail latency observations, and discuss how the discovery of such problems is inhibited by the common focus on throughput performance. We make a case for purposefully re-designing database benchmarking harnesses based on these observations to arrive at faithful characterisations of database performance from multiple important angles.

show abstract

Section: Related Workmentioning

confidence: 91%