ISDI: A New Window-Based Framework for Integrating IoT Streaming Data from Multiple Sources

Tu, Doan Quang; Kayes, A. S. M.; Rahayu, Wenny; Nguyen, Kim Phi Phung

doi:10.1007/978-3-030-15032-7_42

Cited by 16 publications

(18 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, both ontology-based approaches and the Striim engine do not focus on dealing with the timing conflict for time-series data, which is one of the critical issues while integrating data from multiple sources. In this paper, we extend our previous work [28] to integrate IoT streaming data from multiple sources in real-time and deal with the time alignment. In addition, to the best of our knowledge, there is no existing streaming data integration work which considers the issues of information retrieval and data storage, which are our contributions in this paper.…”

Section: Data Integration Techniquesmentioning

confidence: 99%

“…In this section, we introduce our dynamic indexing framework for streaming data from multiple IoT sources, which comprises two main contributions, time-series data compression and time-stamp indexing. In the framework, we also attach the process of time alignment which was introduced in our previous work [28]. Figure 1 illustrates our compression model to store data with the ability of data searching based on users' queries.…”

Section: Proposed Dynamic Indexing Frameworkmentioning

confidence: 99%

“…• The model can extract data from multiple sources continuously through our windowing technique which defines every batch of processed data with a size which equals a window length (a period). This step utilizes our previous work on window extractions [28].…”

Section: Proposed Dynamic Indexing Frameworkmentioning

confidence: 99%

See 2 more Smart Citations

Integration of IoT Streaming Data With Efficient Indexing and Storage Optimization

et al. 2020

Self Cite

View full text Add to dashboard Cite

In the era of IoT, the world of connected experiences is created by the convergence of multiple technologies including real-time analytics, machine learning, and commodity sensors and embedded systems. However, with the proliferation of these IoT technologies and devices, there are challenges in integrating, indexing and managing time-series data from multiple sources to optimize the storage of those data and/or retrieve the information from them in real-time. Many researchers have addressed the data integration issue through developing time-series data compression techniques; however, they focused mainly on the application of integer value compression to IoT data. Moreover, existing work does not focus on the issues of data and information retrieval without decompression. In this paper, we solve these issues by constructing an indexing framework within a lossless compression for floating point time-series data, where an index is based on the time-stamp from the compressed data that facilitates the search for data without full decompression. We conduct several sets of experiments and quantify the performance of our proposed approach. The experimental results, performed on IoT datasets, show a reduction in storage compared with existing compression techniques. The experimental study also demonstrates the capability of time-series data indexing and integration in real-time. INDEX TERMS Data integration, indexing, time-series data compression, floating point compression, decompression, IoT streaming data, window-based compression and integration.

show abstract

Section: Data Integration Techniquesmentioning

confidence: 99%

Section: Proposed Dynamic Indexing Frameworkmentioning

confidence: 99%

See 1 more Smart Citation

Integration of IoT Streaming Data With Efficient Indexing and Storage Optimization

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…(iv) IoT streaming data integration and analysis by machine learning techniques are gaining more interest as most streaming decision models should run in resource-aware environments and detect and react to changes in the environment and many organizations need to deal with massive datasets in different formats coming from multiple sources. The best practice for the performance assessment of how machine learning models is given in [152] while the challenges of IoT streaming data integration are summarized in [153].…”

Section: The Application Layermentioning

confidence: 99%

A Review of Semantic Sensor Technologies in Internet of Things Architectures

Honti

Abonyi

2019

Complexity

View full text Add to dashboard Cite

Intelligent sensors should be seamlessly, securely, and trustworthy interconnected to enable automated high-level smart applications. Semantic metadata can provide contextual information to support the accessibility of these features, making it easier for machines and humans to process the sensory data and achieve interoperability. The unique overview of sensor ontologies according to the semantic needs of the layers of IoT solutions can serve a guideline of engineers and researchers interested in the development of intelligent sensor-based solutions. The explored trends show that ontologies will play an even more essential role in interlinked IoT systems as interoperability and the generation of controlled linkable data sources should be based on semantically enriched sensory data.

show abstract

“…Finally, it is worth mentioning the ISDI algorithm in [37], which is equipped with a windowing routine to analyse and stream data from multiple sources, a timing alignment method and a deduplication algorithm. This algorithm was designed to deal with data streams coming from different sources in the Internet of Things (IoT) systems and can transform multiple data streams, having different attributes, into cleaner datasets suitable for clustering.…”

Section: Introductionmentioning

confidence: 99%

A Clustering System for Dynamic Data Streams Based on Metaheuristic Optimisation

et al. 2019

View full text Add to dashboard Cite

This article presents the Optimised Stream clustering algorithm (OpStream), a novel approach to cluster dynamic data streams. The proposed system displays desirable features, such as a low number of parameters and good scalability capabilities to both high-dimensional data and numbers of clusters in the dataset, and it is based on a hybrid structure using deterministic clustering methods and stochastic optimisation approaches to optimally centre the clusters. Similar to other state-of-the-art methods available in the literature, it uses "microclusters" and other established techniques, such as density based clustering. Unlike other methods, it makes use of metaheuristic optimisation to maximise performances during the initialisation phase, which precedes the classic online phase. Experimental results show that OpStream outperforms the state-of-the-art methods in several cases, and it is always competitive against other comparison algorithms regardless of the chosen optimisation method. Three variants of OpStream, each coming with a different optimisation algorithm, are presented in this study. A thorough sensitive analysis is performed by using the best variant to point out OpStream's robustness to noise and resiliency to parameter changes.

show abstract

ISDI: A New Window-Based Framework for Integrating IoT Streaming Data from Multiple Sources

Cited by 16 publications

References 12 publications

Integration of IoT Streaming Data With Efficient Indexing and Storage Optimization

Integration of IoT Streaming Data With Efficient Indexing and Storage Optimization

A Review of Semantic Sensor Technologies in Internet of Things Architectures

A Clustering System for Dynamic Data Streams Based on Metaheuristic Optimisation

Contact Info

Product

Resources

About