The Embedded IoT Time Series Database for Hybrid Solid-State Storage System

Abstract

IoT time series data is an important form of big data. How to improve the efficiency of storage system is crucial for IoT time series database to store and manage massive IoT time series data from various IoT devices. Mixing NVM and SSD is an effective method to improve the I/O performance of storage systems. However, there are great differences between HDD and NVM or SSD. As a result, NVM and SSD cannot be directly used in the current time series database effectively. We design an IoT time series database with an embedded engine in storage device drivers for the hybrid solid-state storage system consisting of NVM and SSD. The I/O software stack of storing and managing IoT time series data can be shortened to improve the efficiency. Based upon the intrinsic characteristics of IoT time series data and different features of NVM and SSD, a redundancy elimination and compression fusion strategy, a hierarchical management strategy, and a heterogeneous time series data index are designed to improve the efficiency. Finally, a prototype of embedded IoT time series database named TS-NSM is implemented, and YCSB-TS is used to measure the IOPS. The results show that TS-NSM can improve the write IOPS up to 243.6 times and 174.3 times, respectively, compared with InfluxDB and OpenTSDB, and improve the read IOPS up to 10.1 times and 14.4 times, respectively.

A New NVM Device Driver for IoT Time Series Database

“…IoT time series data are not allowed to be updated and deleted randomly, so there are fewer read and write conflicts in IoT time series databases compared with general time series databases. Combined with the partition storage strategy for the IoT time series data that we designed in earlier research [ 8 ], the read and write conflicts can be effectively avoided in the IoT time series database. This provides favorable conditions for improving the concurrency of NVM device drivers for IoT time series databases.…”

“…In order to test and analyze the performance of TS-PMEM, two popular NoSQL time series databases named InfluxDB and OpenTSDB, mentioned in the related work section, were installed with PMEM and TS-PMEM for the comparison. Meanwhile, a new IoT time series database for NVM devices from our group named TS-NSM [ 8 ] was also used for the test, which can skip the file system layer to shorten the I/O software stack for IoT time series databases. In general, there were six prototypes: InfluxDB + PMEM, OpenTSDB + PMEM, TS-NSM + PMEM, InfluxDB + TS-PMEM, OpenTSDB + TS-PMEM, and TS-NSM + TS-PMEM.…”

“… A prototype of new NVM device driver for IoT time series databases named TS-PMEM was implemented. Two popular time series databases and one IoT time series database based on NVM devices from our group [ 8 ] were used for testing by using a general test tool for time series databases named YCSB-TS, and compared with the popular NVM device driver named PMEM from Intel. The results showed that TS-PMEM can increase the write throughput of time series databases by 18.6%, improve the query throughput by 10.6%, reduce the write latency by 8.3%, and decrease the query latency by 6.4% in the maximum compared with the current NVM device driver.…”

Cardinality estimation based on QDSPN for embedded databases under dynamic workload

“…With the proliferation of smart devices and IoT applications, there is a growing need for local data storage and processing capabilities that do not rely on remote servers or clouds. Embedded databases are well-suited to meet these demands, offering high data access speeds, reliability, and security [10][11][12]. However, the resource constraints inherent in embedded systems pose unique challenges for query optimization and cardinality estimation.…”