Gorilla

Abstract

Large-scale internet services aim to remain highly available and responsive in the presence of unexpected failures. Providing this service often requires monitoring and analyzing tens of millions of measurements per second across a large number of systems, and one particularly effective solution is to store and query such measurements in a time series database (TSDB). A key challenge in the design of TSDBs is how to strike the right balance between efficiency, scalability, and reliability. In this paper we introduce Gorilla, Facebook's in-memory TSDB. Our insight is that users of monitoring systems do not place much emphasis on individual data points but rather on aggregate analysis, and recent data points are of much higher value than older points to quickly detect and diagnose the root cause of an ongoing problem. Gorilla optimizes for remaining highly available for writes and reads, even in the face of failures, at the expense of possibly dropping small amounts of data on the write path. To improve query efficiency, we aggressively leverage compression techniques such as delta-of-delta timestamps and XOR'd floating point values to reduce Gorilla's storage footprint by 10x. This allows us to store Gorilla's data in memory, reducing query latency by 73x and improving query throughput by 14x when compared to a traditional database (HBase)-backed time series data. This performance improvement has unlocked new monitoring and debugging tools, such as time series correlation search and more dense visualization tools. Gorilla also gracefully handles failures from a single-node to entire regions with little to no operational overhead.

SGX-Aware Container Orchestration for Heterogeneous Clusters

“…These measures are periodically sent to a central node, in charge of storing and analyzing them. We rely on a time-series database for this task, as they have proved to be an efficient tool to implement queries over moving sliding windows [26]- [28], such as the ones we use in our scheduling policy. Users submit their jobs by specifying the name of the container image (see Figure 2-).…”

A Concept of an In-Memory Database for IoT Sensor Data

“…In addition to these three databases, the time series data base from Facebook Gorilla (Pelkonen et al, 2015) is also from high interest, because it is -a fast, scalable, In-Memory Time Series Database‖ (Pelkonen et al, 2015). Facebook uses this database to storage monitoring data from Facebooks-infrastructure for the past 26 hours and allows the automatic warning in case of anomalies and allows rapid searching of errors.…”

“…The database itself uses for persistent storage the data the Apache HBase database. (Pelkonen et al, 2015) To storage the time serious sensor data, each time series are identified by a unique key. This key is a string which is the only meta data.…”

“…New values are inserted with three elements. The first element is the text of the time series as string, the second the timestamps and the third elements are the sensor values (Pelkonen et al, 2015(Pelkonen et al, , p. 1818. To select these stored data the Beringei Client can be used.…”

“…This paper describes a theoretical concept of a new in memory database, that are based on the advantages of the SAP HANA (Färber et al, 2012) and the open source Gorilla (Pelkonen et al, 2015) database architecture. To understand the used technologies different data compression especially lossy (Goyal et al, 2008) and lossless (Kokovin et al, 2018) compression techniques, have to be discuss and the characteristic of time series data are defined.…”