Real-time ETL in Striim

Pareek, Alok; Khaladkar, Bhushan; Sen, Rajkumar; Onat, Basar; Nadimpalli, Vijay; Lakshminarayanan, Mahadevan

doi:10.1145/3242153.3242157

Cited by 12 publications

(4 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The authors present a view selection methodology with three components: the main memory and cache management; the dynamic selection of MV (materialized views); and a stream ETL process: starts by extracting the instances from the RDF data sources and saving them temporarily in the buffer of inputs. In [31], the authors present a new distributed ETL architecture, Striim developed to collect datasets from different sources and transform them to aggregate without considering the same or different timestamps. The ETL engine runs in a scalable and fault-tolerant cluster of compute nodes.…”

Section: Related Workmentioning

confidence: 99%

Towards a Scalable and Efficient ETL

Gueddoudj,

Chikh

2023

IJCDS

View full text Add to dashboard Cite

Extract, transform, and load (ETL) processes are crucial for building repositories of data from a variety of self-contained sources. Despite their complexity and cost, ETL processes have demonstrated some maturity for traditional, XML, and graph data sources. However the main challenge for ETL processes is double: (1) they do not scale when brought down to managing large and highly varied data sources, involving web-data. (2) the deployment of the target data warehouse in a polystore. The paper reviews various research efforts along this line of research. The paper then proposes a conceptual modeling of these processes using BPMN (Business Process Modeling Notation). These processes are automatically converted to scripts to be implemented within Spark framework. The solution is packaged according a new distributed architecture (Open ETL) that supports both batch and stream processing. To make our new approach more concrete and evaluable, a real case study using the LUBM benchmark, which involves heterogeneous data sources is considered.

show abstract

Section: Related Workmentioning

confidence: 99%

Towards a Scalable and Efficient ETL

Gueddoudj,

Chikh

2023

IJCDS

View full text Add to dashboard Cite

show abstract

“…Boury-Brisset [18], propose the design and implementation of a framework based on scalable multiintelligence data integration services to facilitate the integration of heterogeneous unstructured and structured data. Pareek et al [19] presented a new distributed architecture of ETL Striim developed to collect datasets from different sources. The ETL engine runs on a scalable and fault-tolerant cluster of compute nodes.…”

Section: Related Workmentioning

confidence: 99%

Os-ETL: A High-Efficiency, Open-Scala Solution for Integrating Heterogeneous Data in Large-Scale Data Warehousing

Gueddoudj,

Chikh,

Attia

2023

ISI

View full text Add to dashboard Cite

The surge in data volume necessitates the integration of Resource Description Framework (RDF) data within corporate environments. While Extract, Transform, Load (ETL) processes exhibit proficiency with conventional data sources, their scalability diminishes when applied to large and highly varied data sources, inclusive of RDF data. The latter constitutes a wealth of knowledge that, when harnessed via data warehouse technology, can augment corporate value in a fiercely competitive milieu. The advent of platforms like polystore offers opportunities for advanced hardware deployment. ETL processes necessitate two crucial phases: Partitioning and data allocation. Concurrently, the scientific community is spurred to innovate ETL processes that support real-time analytics. This study proposes a novel architecture for ETL processes, named Open-Scala-ETL (Os-ETL). Equipped with a method for deploying a data warehouse based on a polystore, Os-ETL enables real-time analysis. The primary objective of the Os-ETL solution is to resolve the complexities in deploying a graph structure data warehouse on a polystore-a process that involves partitioning and data allocation. Os-ETL is a distributed solution that supports both batch and streaming processing using the Spark framework. Scala scripts are executed within this framework to partition RDF graphs and distribute the resultant fragments across various sites. The implementation of Os-ETL is based on Apache Spark, with ETL deployment on a Spark SQL polystore. This solution empowers companies with data warehouse technology to improve performance, scalability, and latency between a data warehouse and its data sources. The approach has been assessed and validated using largescale, heterogeneous data, encompassing the LUBM benchmark, CSV files, an Oracle database, and a Neo4j graph database. The results corroborate its superior performance in terms of scalability and optimization.

show abstract

“…Lastly, data output involves loading transformed data immediately into target systems for further analysis, storage, or visualization. Output systems may include databases, data warehouses, or other analytics platforms, depending on the specific requirements of the application [19]. In this section, we will mention some works related to our article that served as either direct or indirect inspiration and solved similar problems or utilized the same technologies as our team.…”

Section: Real-time Data Processing Toolsmentioning

confidence: 99%

A Modular Framework for Data Processing at the Edge: Design and Implementation

Urblik,

Kajati,

Papcun

et al. 2023

Sensors

View full text Add to dashboard Cite

There is a rapid increase in the number of edge devices in IoT solutions, generating vast amounts of data that need to be processed and analyzed efficiently. Traditional cloud-based architectures can face latency, bandwidth, and privacy challenges when dealing with this data flood. There is currently no unified approach to the creation of edge computing solutions. This work addresses this problem by exploring containerization for data processing solutions at the network’s edge. The current approach involves creating a specialized application compatible with the device used. Another approach involves using containerization for deployment and monitoring. The heterogeneity of edge environments would greatly benefit from a universal modular platform. Our proposed edge computing-based framework implements a streaming extract, transform, and load pipeline for data processing and analysis using ZeroMQ as the communication backbone and containerization for scalable deployment. Results demonstrate the effectiveness of the proposed framework, making it suitable for time-sensitive IoT applications.

show abstract

Real-time ETL in Striim

Cited by 12 publications

References 10 publications

Towards a Scalable and Efficient ETL

Towards a Scalable and Efficient ETL

Os-ETL: A High-Efficiency, Open-Scala Solution for Integrating Heterogeneous Data in Large-Scale Data Warehousing

A Modular Framework for Data Processing at the Edge: Design and Implementation

Contact Info

Product

Resources

About