Data transformation as a means towards dynamic data storage and polyglot persistence

Vanhove, Thomas; Sebrechts, Merlijn; Seghbroeck, Gregory Van; Wauters, Tim; Volckaert, Bruno; Turck, Filip De

doi:10.1002/nem.1976

Cited by 5 publications

(12 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, research has been performed towards the automated transformation of both schema and data between different storage technologies. The algorithm proposed in [15] detects the format of the dataset from the original data storage and transforms it in such a way that it can benefit from the characteristics of the target data storage technology, allowing the data to be presented in any supported technology of their choice. This transformation operation is performed by the data analysis platform.…”

Section: Data Analysis Platformmentioning

confidence: 99%

City of Things: Enabling Resource Provisioning in Smart Cities

et al. 2018

Self Cite

View full text Add to dashboard Cite

Abstract-In the last years, traffic over wireless networks has been increasing exponentially due to the impact of Internet of Things (IoT). IoT is transforming a wide range of services in different domains of urban life, such as environmental monitoring, home automation and public transportation. The so-called Smart City applications will introduce a set of stringent requirements, such as low latency and high mobility, since services must be allocated and instantiated on-demand simultaneously close to multiple devices at different locations. Efficient resource provisioning functionalities are needed to address these demanding constraints introduced by Smart City applications while minimizing resource costs and maximizing Quality of Service (QoS). In this article, the City of Things (CoT) framework is presented, which provides not only data collection and analysis functionalities but also automated resource provisioning mechanisms for future Smart City applications. CoT is deployed as a Smart City testbed in Antwerp (Belgium) that allows researchers and developers to easily setup and validate IoT experiments. A Smart City use case based on Air Quality Monitoring through the deployment of air quality sensors in moving cars has been presented showing the full applicability of the CoT framework for a flexible and scalable resource provisioning in the Smart City ecosystem.

show abstract

Section: Data Analysis Platformmentioning

confidence: 99%

City of Things: Enabling Resource Provisioning in Smart Cities

et al. 2018

Self Cite

View full text Add to dashboard Cite

show abstract

“…This approach however entails a number of limitations in terms of (i) usability, as end-users need to formulate queries using a formal query language, (ii) costs of creating and maintaining both ontology and mappings, and (iii) efficiency of both the translation process and the execution of the resulting queries. In [9] the usage of multiple storage technologies in a single application is regarded as polyglot persistence. When optimally applied, polyglot persistence boosts application performance by storing each different data type into the data store technology best suited for it.…”

Section: Related Workmentioning

confidence: 99%

“…those storing their data in one relational data store. The mechanism conceived in [9] is able to transform schema and data between a source and a target data store without application downtime and avoiding code changes, as client applications are not required to change the query language they use, thanks to an ever running live query transformation. The transformation process is based on the Lambda Architecture [3]: the batch layer is in charge of transforming a snapshot of the data available in the source data store, while the speed layer performs transformations on the stream of new queries issued after the snapshot was taken.…”

Section: Related Workmentioning

confidence: 99%

“…This component builds further on the data transformation mechanism conceived by Vanhove et al in [9], by turning it into an iterative process.…”

Section: Dynamic Read-optimizationmentioning

confidence: 99%

“…The proposed approach extends existing methods of data transformation for polyglot persistence with no application downtime [9], and automatic schema denormalization based on query workload analysis [10].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Dynamic data transformation for low latency querying in big data systems

Ordonez-Ante¹,

Vanhove²,

Seghbroeck³

et al. 2017

2017 IEEE International Conference on Big Data (Big Data)

Self Cite

View full text Add to dashboard Cite

Abstract-Big data storage technologies inherently entail high latency characteristics, preventing users from performing efficient ad-hoc querying and interactive visualization on large and distributed datasets. Most of the existing approaches addressing this issue thrive on de-normalization of the static data schema and creation of application specific (i.e. hardcoded) materialized views, which certainly reduce data access latency but at the expense of flexibility. In this regard, this paper proposes an approach that relies on an iterative process of data transformation intended to generate read-optimized data schemas. The transformation process is able to automatically identify optimization opportunities (e.g. materialized views, missing indexes), by analyzing the original data schema and the record of queries issued by users and client applications against the data set. An experimental evaluation of the proposed approach evidences a significant reduction in the query latency, ranging from 81.60% to 99.99%.

show abstract

A Workload-Driven Approach for View Selection in Large Dimensional Datasets

et al. 2020

Self Cite

View full text Add to dashboard Cite

The information explosion the world has witnessed in the last two decades has forced businesses to adopt a data-driven culture for them to be competitive. These data-driven businesses have access to countless sources of information, and face the challenge of making sense of overwhelming amounts of data in a efficient and reliable manner, which implies the execution of readintensive operations. In the context of this challenge, a framework for the dynamic read-optimization of large dimensional datasets has been designed, and on top of it a workload-driven mechanism for automatic materialized view selection and creation has been developed. This paper presents an extensive description of this mechanism, along with a proof-of-concept implementation of it and its corresponding performance evaluation. Results show that the proposed mechanism is able to derive a limited but comprehensive set of views leading to a drop in query latency ranging from 80% to 99.99% at the expense of 13% of the disk space used by the base dataset. This way, the devised mechanism enables speeding up query execution by building materialized views that match the actual demand of query workloads.

show abstract

Data transformation as a means towards dynamic data storage and polyglot persistence

Cited by 5 publications

References 32 publications

City of Things: Enabling Resource Provisioning in Smart Cities

City of Things: Enabling Resource Provisioning in Smart Cities

Dynamic data transformation for low latency querying in big data systems

A Workload-Driven Approach for View Selection in Large Dimensional Datasets

Contact Info

Product

Resources

About