NagareDB: A Resource-Efficient Document-Oriented Time-Series Database

Abstract

The recent great technological advance has led to a broad proliferation of Monitoring Infrastructures, which typically keep track of specific assets along time, ranging from factory machinery, device location, or even people. Gathering this data has become crucial for a wide number of applications, like exploration dashboards or Machine Learning techniques, such as Anomaly Detection. Time-Series Databases, designed to handle these data, grew in popularity, becoming the fastest-growing database type from 2019. In consequence, keeping track and mastering those rapidly evolving technologies became increasingly difficult. This paper introduces the holistic design approach followed for building NagareDB, a Time-Series database built on top of MongoDB—the most popular NoSQL Database, typically discouraged in the Time-Series scenario. The goal of NagareDB is to ease the access to three of the essential resources needed to building time-dependent systems: Hardware, since it is able to work in commodity machines; Software, as it is built on top of an open-source solution; and Expert Personnel, as its foundation database is considered the most popular NoSQL DB, lowering its learning curve. Concretely, NagareDB is able to outperform MongoDB recommended implementation up to 4.7 times, when retrieving data, while also offering a stream-ingestion up to 35% faster than InfluxDB, the most popular Time-Series database. Moreover, by relaxing some requirements, NagareDB is able to reduce the disk space usage up to 40%.

Introducing Polyglot-Based Data-Flow Awareness to Time-Series Data Stores

“…• NagareDB is a Time-series database built on top of Mon-goDB, which lowers its learning curve. Its data model, built on top of MongoDB's document-oriented data model, follows a column-oriented approximation, as data columns are embedded inside JSON-like documents [7]. NagareDB inherits most of MongoDB's features, including its query language, its free and straight-forward horizontal scalability.…”

“…NagareDB inherits most of MongoDB's features, including its query language, its free and straight-forward horizontal scalability. It is a free, competitive alternative to popular and enterprise-licensed time-series databases [7], both in terms of querying and ingestion performance-however not always with a consistent or remarkable speed-up, sometimes falling behind.…”

“…Thus, JSON-like documents, the basic data structure of the underlying database, are intended to store data in a columnar shape, following a schema-fixed approach. The specific data embedding mechanisms that DM2 follows are extensively detailed in NagareDB's presentation research study as, actually, the original data model of NagareDB is equivalent to this research's DM2 [7]. In disk, it is organized following the natural time line, according to data arrival order from DM1: All sensor's data from a given day will be placed adjacently.…”

“…However, since it is the baseline data model of NagareDB, it will also participate in yearly queries, in order to obtain further insights and behaviour differences. Last, NagareDB is able to use limitedprecision data types, allowing up to 40% of disk usage while providing further speedup [7]. However, as this behaviour does not affect the effectiveness of the polyglot mechanisms, this benchmark only includes full-precision data types, in order to avoid repetitive or trivial results.…”

“…In order to evaluate its performance, we materialize our approach in an alternative implementation of NagareDB, a Time-Series database [7] built on top of MongoDB, the most popular open-source NoSQL database [6]. With this evaluation approach we aim (1) to demonstrate that the proposed technique is capable of outperforming popular and mainstream approaches, and (2) to illustrate that it is possible to improve and adapt already-existent databases, in order to cope with demanding specific-purpose scenarios, relieving the need of developing further database management systems (DBMS) from scratch.…”