Data Load Distribution by Semi Real Time Data Warehouse

Abstract

Today many organizations used data warehouse for strategic decision making. Today's realtime business stresses the potential to process increasingly volumes of data at very high speed in order to stay competitive in market. Data Warehouse is populated by data extraction, transformation and loading from different data sources by software utilities called ETL (Extraction, transformation & loading). ETL process is a time consuming process as it has to process large volume of data. ETL processes must have certain completion time window and ETL process must have to finish within this time window. In this paper we discusses a technique to distribute the volume of data to be extracted, transformed and loaded into data warehouse by merging both conventional and real-time techniques, so ETL process finishes its job within its time window by utilizing ETL idle time.

An Innovative Method to Extract Data in a Real-time Data Warehousing Environment

“…Regarding to meeting real-time data warehousing requirement, Bruckner et al [26], Naeem et al [27], Javed, M. and Nawaz, A. [28], YiChuan and Yao [29], MadeSukarsa et al [30], Jain et al [13], Jia et al [31], Obali et al [32], Mao et al [33], Li and Mao [20] and Muddasir and Raghuveer [9] deals with low latency. Further, just Bruckner et al [26] and Viana et al [34] deal with availability requirements.…”

Change Data Capture on OLTP Staging Area for Nearly Real Time Data Warehouse Base on Database Trigger

“…Data source from various OLTP processed through the various stages that consist of Extract, Transform and Loading (ETL). ETL is built on a tier that is placed between the source data and DWH and known also as a staging area [2]. Extract part relied on to take data from multiple sources within a specific time period to be taken to DWH.…”