Incremental maintenance of aggregate and outerjoin expressions

Gupta, Himanshu; Mumick, Inderpal Singh

doi:10.1016/j.is.2004.11.011

Cited by 38 publications

(28 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In our experiments, we use use z = 2 unless otherwise noted. The incremental maintenance algorithm used in our experiments is the "change-table" or "delta-table" method used in numerous works in incremental maintenance [22,23,27]. In all of the applications, the updates are kept in memory in a temporary table, and we discount this loading time from our experiments.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Stale view cleaning

et al. 2015

View full text Add to dashboard Cite

Materialized views (MVs), stored pre-computed results, are widely used to facilitate fast queries on large datasets. When new records arrive at a high rate, it is infeasible to continuously update (maintain) MVs and a common solution is to defer maintenance by batching updates together. Between batches the MVs become increasingly stale with incorrect, missing, and superfluous rows leading to increasingly inaccurate query results. We propose Stale View Cleaning (SVC) which addresses this problem from a data cleaning perspective. In SVC, we efficiently clean a sample of rows from a stale MV, and use the clean sample to estimate aggregate query results. While approximate, the estimated query results reflect the most recent data. As sampling can be sensitive to long-tailed distributions, we further explore an outlier indexing technique to give increased accuracy when the data distributions are skewed. SVC complements existing deferred maintenance approaches by giving accurate and bounded query answers between maintenance. We evaluate our method on a generated dataset from the TPC-D benchmark and a real video distribution application. Experiments confirm our theoretical results: (1) cleaning an MV sample is more efficient than full view maintenance, (2) the estimated results are more accurate than using the stale MV, and (3) SVC is applicable for a wide variety of MVs.

show abstract

Section: Methodsmentioning

confidence: 99%

“…Incremental View Maintenance (IVM): We apply incremental view maintenance (change-table based maintenance [22,23,27]) to the full view. SVC+AQP: We maintain a sample of the materialized view using SVC and estimate the result with AQP-style estimation technique.…”

Section: Metrics and Evaluationmentioning

confidence: 99%

Stale view cleaning

et al. 2015

View full text Add to dashboard Cite

show abstract

“…Contrary to our setting, these algorithms are designed to operate in a batch, off-line fashion. Work in materialized views refreshment [9,10,23,24] is also relevant, but orthogonal to our setting. The crucial decision concerns whether a view can be updated given a delta set of updates.…”

Section: Related Workmentioning

confidence: 99%

Supporting Streaming Updates in an Active Data Warehouse

Polyzotis

Skiadopoulos

Vassiliadis

et al. 2007

2007 IEEE 23rd International Conference on Data Engineering

View full text Add to dashboard Cite

show abstract

“…Work in materialized views refreshment [12,13,24,25] is orthogonal to our setting. In [13] the authors describe materialized views, their applications, and the problems and techniques for their maintenance.…”

Section: Related Workmentioning

confidence: 99%

“…In [13] the authors describe materialized views, their applications, and the problems and techniques for their maintenance. Novel techniques and an up-to-date survey of related work in the field are presented in [12]. Materialized views refreshment fits orthogonally with our on-line refreshment technique, since we can treat each ETL queue as a black-box process.…”

Section: Related Workmentioning

confidence: 99%

ETL queues for active data warehousing

Karakasidis

Vassiliadis

Pitoura

2005

Proceedings of the 2nd International Workshop on Information Quality in Information Systems

View full text Add to dashboard Cite

Traditionally, the refreshment of data warehouses has been performed in an off-line fashion. Active Data Warehousing refers to a new trend where data warehouses are updated as frequently as possible, to accommodate the high demands of users for fresh data. In this paper, we propose a framework for the implementation of active data warehousing, with the following goals: (a) minimal changes in the software configuration of the source, (b) minimal overhead for the source due to the active nature of data propagation, (c) the possibility of smoothly regulating the overall configuration of the environment in a principled way. In our framework, we have implemented ETL activities over queue networks and employ queue theory for the prediction of the performance and the tuning of the operation of the overall refreshment process. Due to the performance overheads incurred, we explore different architectural choices for this task and discuss the issues that arise for each of them.

show abstract

Incremental maintenance of aggregate and outerjoin expressions

Cited by 38 publications

References 19 publications

Stale view cleaning

Stale view cleaning

Supporting Streaming Updates in an Active Data Warehouse

ETL queues for active data warehousing

Contact Info

Product

Resources

About