Enhancing Traditional Data Warehousing Architectures with Real-Time Capabilities

Cuzzocrea, Alfredo; Ferreira, Nickerson; Furtado, Pedro

doi:10.1007/978-3-319-08326-1_46

Cited by 7 publications

(3 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The latest trend of data warehousing is to support big data, and offer real-time/right-time capability, e.g., [3,24]. The emergence of the cloud computing technologies, such as MapReduce, makes it feasible for ETL to process large-scale datasets.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

An ETL optimization framework using partitioning and parallelization

Liu

Iftikhar

2015

Proceedings of the 30th Annual ACM Symposium on Applied Computing

View full text Add to dashboard Cite

Extract-Transform-Load (ETL) handles large amounts of data and manages workload through dataflows. ETL dataflows are widely regarded as complex and expensive operations in terms of time and system resources. In order to minimize the time and the resources required by ETL dataflows, this paper presents an optimization framework using partitioning and parallelization. The framework first partitions an ETL dataflow into multiple execution trees according to the characteristics of ETL constructs, then within an execution tree pipelined parallelism and shared cache are used to optimize the partitioned dataflow. Furthermore, multi-threading is used in component-based optimization. The experimental results show that the proposed framework can achieve 4.7 times faster than the ordinary ETL dataflows (without using the proposed partitioning and optimization methods), and is comparable to the similar ETL tools.

show abstract

Section: Related Workmentioning

confidence: 99%

“…The thread will be blocked if there is no space available in the queue. The shared cache is processed sequentially through all the activity threads (see line [1][2][3][4][5][6][7][8][9][10][11]. For each component, it has to process m shared caches in a sequential order.…”

Section: Inside Execution Tree Parallelizationmentioning

confidence: 99%

An ETL optimization framework using partitioning and parallelization

Liu

Iftikhar

2015

Proceedings of the 30th Annual ACM Symposium on Applied Computing

View full text Add to dashboard Cite

show abstract

“…The latest trend of data warehousing is to support big data, and offer real-time/right-time capability, e.g., [3,27]. The emergence of the cloud computing technologies, such as MapReduce [4], makes it feasible for ETL to process large-scale data on many nodes.…”

Section: Related Workmentioning

confidence: 99%

Two-level Data Staging ETL for Transaction Data

Liu

2014

Preprint

View full text Add to dashboard Cite

In data warehousing, Extract-Transform-Load (ETL) extracts the data from data sources into a central data warehouse regularly for the support of business decision-makings. The data from transaction processing systems are featured with the high frequent changes of insertion, update, and deletion. It is challenging for ETL to propagate the changes to the data warehouse, and maintain the change history. Moreover, ETL jobs typically run in a sequential order when processing the data with dependencies, which is not optimal, e.g., when processing early-arriving data. In this paper, we propose a two-level data staging ETL for handling transaction data. The proposed method detects the changes of the data from transactional processing systems, identifies the corresponding operation codes for the changes, and uses two staging databases to facilitate the data processing in an ETL process. The proposed ETL provides the "onestop" method for fast-changing, slowly-changing and early-arriving data processing.

show abstract

A big data perspective of current ETL techniques

Phanikanth

Sudarsan

2016

2016 International Conference on Advances in Computing and Communication Engineering (ICACCE)

View full text Add to dashboard Cite

Enhancing Traditional Data Warehousing Architectures with Real-Time Capabilities

Cited by 7 publications

References 4 publications

An ETL optimization framework using partitioning and parallelization

An ETL optimization framework using partitioning and parallelization

Two-level Data Staging ETL for Transaction Data

A big data perspective of current ETL techniques

Contact Info

Product

Resources

About