Using REO on ETL conceptual modelling

Abstract

The formalization of software patterns has proven to be very useful in software developing, improving systems communication, data interchange across platforms, and simplifying the integration of processes and data flows. Populating a data warehouse (ETL) is often a very complex task demanding significant computational resources, involving not only large volumes of data that must be processed but also undesirable change of business requirements. All of this leads frequently to reuse significant parts of other ETL implementations, adapting data structures and processes to comply with new requirements. Additionally, we believe that it's necessary a more simply and reliable approach for ETL conceptual modelling covering the "lack of mature" of this important part of ETL development. In this paper we explored a new approach to ETL conceptual modelling using the Reo coordination language, trying to evaluate its adequacy and expressiveness on the coordination of ETL tasks. A pattern-based approach was designed to map typical operations used in real world ETL scenarios from an initial Reo specification. For demonstration purposes, we present and discuss as two case studies, a slowly changing dimension and a surrogated key pipelining processes.

BigDimETL: ETL for Multidimensional Big Data

“…In other hand, several studies focused on the development of ETL functionalities model which are considered as important sub-processes of ETL. For example, [17] designs CDC functionality (Changing Data Capture) as colored network (CPN) describing and presenting the behavior of ETL processes, and [15] presents SCD (slowly Changing Dimension) and SKP (Surrogate key Pipeline) through a coordination Reo language to improve the synchronization between the entities that interact with each other. Today, Information System researchers are facing a big explosion of web data, especially with social media (Facebook, Twitter, Youtube,).…”

Modeling and Supporting ETL Processes via a Pattern-Oriented, Task-Reusable Framework

“…In spite of the expressiveness and utility capabilities of BPMN especially at the initial stages of development, several difficulties emerge when we need to generate a specific executable model from a previous conceptual model. However, the REO coordination language [14] takes us to other level on process formalization, allowing for more "task independent" models and for the simulation of model designed. If we focus primarily at the coordination level representation rather to tasks internal composition, we can represent ETL patterns as black boxes, avoiding to show numerous details, tasks and ETL artefacts, at a technical level that any user with basic ETL knowledge and skills will be able to interpret and understand and ETL system model from scratch.…”

On the conceptualization of ETL patterns

“…More recently, Akkaoui et al [11] provided a framework that allows for the translation of abstract BPMN models to its concrete execution in a target ETL tool. In some other works we already explored the use of ETL patterns with BPMN for ETL conceptual modelling [12]- [16]. Although the experiments done with BPMN, we have also explore the use of Reo [16] in the representation of ETL patterns.…”

“…In some other works we already explored the use of ETL patterns with BPMN for ETL conceptual modelling [12]- [16]. Although the experiments done with BPMN, we have also explore the use of Reo [16] in the representation of ETL patterns. When compared to BPMN, Reo provides a more concrete and formal way to define ETL processes, avoiding the lack of expressiveness of the BPMN notation that is very useful to communicate with non-technical users but difficult to convert to ETL processes execution primitives.…”