A Novel Approach to Handle Huge Data for Refreshment Anomalies in Near Real-Time ETL Applications

Muddasir, N. Mohammed; Raghuveer, K.

doi:10.1007/978-981-15-4032-5_50

Cited by 3 publications

(1 citation statement)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our previous work on refreshment anomalies [17] We had implemented the idea of stalled query for getting updates. As compared to this work in our previous work the updates were stored in temporary tables, here we have a novel approach to calculate the threshold before stalling, later to have freshness interval to determine whether it's required to stall all queries on only some finally to limit the time given for performing the updates.…”

Section: Related Workmentioning

confidence: 99%

Towards Handling Incremental Load for Anomalies in Near Real Time Data Warehouse

Muddasir¹,

Raghuveer²,

Dayanand³

2020

Wseas Transactions on Systems and Control

Self Cite

View full text Add to dashboard Cite

Refreshment anomalies occur in a data warehousing environment while performing Extract Transform and Load (ETL) to get the data for analysis from sources. There could be several reasons for the anomalies like not able to capture the delta on time, system time out, duplicate entries due to outer join operations and many more. Once anomalies are detected the compensation operation is executed to get the data that was missing into the data warehouse. In this work we would like to analyze scenario where it is necessary to perform incremental loads based on priority in an ongoing data warehouse maintenance work. The work proposes a novel approach to decide on when to perform ETL so that refreshment anomalies do not occur and to maintain integrity of data such that analytics queries always provide right information to the analyst. Two novelties have been discussed in this work one is to have a threshold before compensation of updates and two is while performing compensation updates prioritize the query with less freshness interval to have more time limits for the updates to be completed.

show abstract

Section: Related Workmentioning

confidence: 99%