IoT streamed data handling model using delta encoding

Jindal, Rajni; Kumar, Neetesh; Patidar, Sanjay

doi:10.1002/dac.5243

Cited by 3 publications

(4 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the absence of delta being specified in the metadata, a default value of 1 is assumed. In the same direction as other works such as [45], delta encoding typically involves only horizontal differences. An example of a delta value as found within the metadata is presented in Listing 29.…”

Section: : Deltamentioning

confidence: 99%

Yet Another Compact Time Series Data Representation Using CBOR Templates (YACTS)

Sebastian

Lozada-Martínez

Papadopoulos

et al. 2023

Sensors

View full text Add to dashboard Cite

The Internet of Things (IoT) technology is growing rapidly, while the IoT devices are being deployed massively. However, interoperability with information systems remains a major challenge for this accelerated device deployment. Furthermore, most of the time, IoT information is presented as Time Series (TS), and while the majority of the studies in the literature focus on the prediction, compression, or processing of TS, no standardized representation format has emerged. Moreover, apart from interoperability, IoT networks contain multiple constrained devices which are designed with limitations, e.g., processing power, memory, or battery life. Therefore, in order to reduce the interoperability challenges and increase the lifetime of IoT devices, this article introduces a new format for TS based on CBOR. The format exploits the compactness of CBOR by leveraging delta values to represent measurements, employing tags to represent variables, and utilizing templates to convert the TS data representation into the appropriate format for the cloud-based application. Moreover, we introduce a new refined and structured metadata to represent additional information for the measurements, then we provide a Concise Data Definition Language (CDDL) code to validate the CBOR structures against our proposal, and finally, we present a detailed performance evaluation to validate the adaptability and the extensibility of our approach. Our performance evaluation results show that the actual data sent by IoT devices can be reduced by between 88% and 94% compared to JavaScript Object Notation (JSON), between 82% and 91% compared to Concise Binary Object Representation (CBOR) and ASN.1, and between 60% and 88% compared to Protocol buffers. At the same time, it can reduce Time-on-Air by between 84% and 94% when a Low PowerWide Area Networks (LPWAN) technology such as LoRaWAN is employed, leading to a 12-fold increase in battery life compared to CBOR format or between a 9-fold and 16-fold increase when compared to Protocol buffers and ASN.1, respectively. In addition, the proposed metadata represent an additional 0.5% of the overall data transmitted in cases where networks such as LPWAN or Wi-Fi are employed. Finally, the proposed template and data format provide a compact representation of TS that can significantly reduce the amount of data transmitted containing the same information, extend the battery life of IoT devices, and improve their lifetime. Moreover, the results show that the proposed approach is effective for different data types and it can be integrated seamlessly into existing IoT systems.

show abstract

Section: : Deltamentioning

confidence: 99%

Yet Another Compact Time Series Data Representation Using CBOR Templates (YACTS)

Sebastian

Lozada-Martínez

Papadopoulos

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…All these lossless algorithms compress data and can decode the reduced algorithm back to its original size. Some examples of lossless algorithms include Delta Encoding, LZ77, LZ78, Huffman coding, and Run Length Encoding [23,24].  Lossy data reduction techniques [10,[25][26][27]: these focus on reducing data by discarding some details considered less relevant to the analysis or less noticeable to human perception, and thus achieve higher compression ratios than other methods.…”

Section: Proposed Taxonomymentioning

confidence: 99%

mentioning

confidence: 99%

Analyzing Data Reduction Techniques: An Experimental Perspective

Fernandes,

Carvalho,

Pereira

et al. 2024

Applied Sciences

View full text Add to dashboard Cite

The exponential growth in data generation has become a ubiquitous phenomenon in today’s rapidly growing digital technology. Technological advances and the number of connected devices are the main drivers of this expansion. However, the exponential growth of data presents challenges across different architectures, particularly in terms of inefficient energy consumption, suboptimal bandwidth utilization, and the rapid increase in data stored in cloud environments. Therefore, data reduction techniques are crucial to reduce the amount of data transferred and stored. This paper provides a comprehensive review of various data reduction techniques and introduces a taxonomy to classify these methods based on the type of data loss. The experiments conducted in this study include distinct data types, assessing the performance and applicability of these techniques across different datasets.

show abstract

“…After the chunks are created, if the compression is enabled, the TimescaleDB compression engine compresses the chunks to reduce the storage overhead ❹. TimescaleDB compression engine consists of several compression algorithms for different data types. For example, if the chunk data is an integer, it is compressed with delta encoding [24], simple-8b [25], delta of delta encoding [26], and run-length encoding [27]. Also, in the case of a floating point, it is compressed with XOR-based compression [26].…”

Section: ) Time Series Databasesmentioning

confidence: 99%

Analyzing I/O Characteristics of Time-Series Data Using High Performance Storage Devices

Lee,

Son,

Kim

2023

IEEE Access

View full text Add to dashboard Cite

As the importance of data increases, data is continuously collected from diverse sources such as sensors, IoT devices, and edge computing devices. To manage these continuously monitored data, it is often organized chronologically with time which is referred as time-series data. By managing the data using time, data from different streams can be analyzed in a comprehensive manner with an identical index which is time. However, due to the unique characteristics of time-series data, it is essential for the underlying database systems to understand the characteristics of the time-series data. To handle this, time-series database systems, which specially target time-series data, are emerging. These database systems have different performance characteristics due to the unique characteristics of the data which should be investigated to efficiently store and analyze the data. In this paper, we analyze the time-series database from the perspective of I/O using various storage devices from HDD, SATA and NVMe SSD. First, we analyze the I/O characteristics such as runtime, throughput and size of total requests using various storage devices. In addition, we analyze the effect of unique time-series database features such as data chunk interval, compression and number of workers. Our analysis results show that adapting high-performance devices can greatly improve the performance of the database by up to 33.22×.

show abstract

IoT streamed data handling model using delta encoding

Cited by 3 publications

References 34 publications

Yet Another Compact Time Series Data Representation Using CBOR Templates (YACTS)

Yet Another Compact Time Series Data Representation Using CBOR Templates (YACTS)

Analyzing Data Reduction Techniques: An Experimental Perspective

Analyzing I/O Characteristics of Time-Series Data Using High Performance Storage Devices

Contact Info

Product

Resources

About