DBToaster: higher-order delta processing for dynamic, frequently fresh views

Koch, Christoph; Ahmad, Yanif; Kennedy, Oliver; Nikolić, Miloš; Nötzli, Andres; Lupei, Daniel; Shaikhha, Amir

doi:10.1007/s00778-013-0348-4

Cited by 73 publications

(114 citation statements)

References 42 publications

Supporting

Mentioning

112

Contrasting

Unclassified

Order By: Relevance

“…In our experiments, we use use z = 2 unless otherwise noted. The incremental maintenance algorithm used in our experiments is the "change-table" or "delta-table" method used in numerous works in incremental maintenance [22,23,27]. In all of the applications, the updates are kept in memory in a temporary table, and we discount this loading time from our experiments.…”

Section: Methodsmentioning

confidence: 99%

“…The problem with eager maintenance is that every view created by an analyst places a bottleneck on incoming transactions. There has been significant research on fast MV maintenance algorithms, most recently DBToaster [27] which uses SQL query compilation and higher-order maintenance. However, even with these optimizations, some materialized views are computationally difficult to incrementally maintain.…”

Section: Motivation and Examplementioning

confidence: 99%

“…Incremental View Maintenance (IVM): We apply incremental view maintenance (change-table based maintenance [22,23,27]) to the full view. SVC+AQP: We maintain a sample of the materialized view using SVC and estimate the result with AQP-style estimation technique.…”

Section: Metrics and Evaluationmentioning

confidence: 99%

“…However, as with any precomputation or caching, the key challenge in using MVs is maintaining their freshness as base data changes. While there has been substantial work in incremental maintenance of MVs [9,27], eager maintenance (i.e., immediately applying updates) is not always feasible.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Stale view cleaning

et al. 2015

View full text Add to dashboard Cite

Materialized views (MVs), stored pre-computed results, are widely used to facilitate fast queries on large datasets. When new records arrive at a high rate, it is infeasible to continuously update (maintain) MVs and a common solution is to defer maintenance by batching updates together. Between batches the MVs become increasingly stale with incorrect, missing, and superfluous rows leading to increasingly inaccurate query results. We propose Stale View Cleaning (SVC) which addresses this problem from a data cleaning perspective. In SVC, we efficiently clean a sample of rows from a stale MV, and use the clean sample to estimate aggregate query results. While approximate, the estimated query results reflect the most recent data. As sampling can be sensitive to long-tailed distributions, we further explore an outlier indexing technique to give increased accuracy when the data distributions are skewed. SVC complements existing deferred maintenance approaches by giving accurate and bounded query answers between maintenance. We evaluate our method on a generated dataset from the TPC-D benchmark and a real video distribution application. Experiments confirm our theoretical results: (1) cleaning an MV sample is more efficient than full view maintenance, (2) the estimated results are more accurate than using the stale MV, and (3) SVC is applicable for a wide variety of MVs.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Motivation and Examplementioning

confidence: 99%

Section: Metrics and Evaluationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Stale view cleaning

et al. 2015

View full text Add to dashboard Cite

show abstract

“…And useless MVs shall be removed for optimization purpose, and maximum queries will be satisfied with the help of MVs (Koch, Ahmad et al 2014).…”

Section: B Creation and Maintenance Of Child Viewsmentioning

confidence: 99%

Data Cleaning In Data Warehouse: A Survey of Data Pre-processing Techniques and Tools

Fatima¹,

Nazir²,

Khan³

2017

IJITCS

View full text Add to dashboard Cite

Abstract-A Data Warehouse is a computer system designed for storing and analyzing an organization's historical data from day-to-day operations in Online Transaction Processing System (OLTP). Usually, an organization summarizes and copies information from its operational systems to the data warehouse on a regular schedule and management performs complex queries and analysis on the information without slowing down the operational systems. Data need to be pre-processed to improve quality of data, before storing into data warehouse. This survey paper presents data cleaning problems and the approaches in use currently for preprocessing. To determine which technique of preprocessing is best in what scenario to improve the performance of Data Warehouse is main goal of this paper. Many techniques have been analyzed for data cleansing, using certain evaluation attributes and tested on different kind of data sets. Data quality tools such as YALE, ALTERYX, and WEKA have been used for conclusive results to ready the data in data warehouse and ensure that only cleaned data populates the warehouse, thus enhancing usability of the warehouse. Results of paper can be useful in many future activities like cleansing, standardizing, correction, matching and transformation. This research can help in data auditing and pattern detection in the data.

show abstract

Incremental $$\lambda $$ -Calculus in Cache-Transfer Style

Giarrusso

Régis-Gianas

Schuster

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Incremental computation requires propagating changes and reusing intermediate results of base computations. Derivatives, as produced by static differentiation [7], propagate changes but do not reuse intermediate results, leading to wasteful recomputation. As a solution, we introduce conversion to Cache-Transfer-Style, an additional program transformations producing purely incremental functional programs that create and maintain nested tuples of intermediate results. To prove CTS conversion correct, we extend the correctness proof of static differentiation from STLC to untyped λ-calculus via step-indexed logical relations, and prove sound the additional transformation via simulation theorems.To show ILC-based languages can improve performance relative to from-scratch recomputation, and that CTS conversion can extend its applicability, we perform an initial performance case study. We provide derivatives of primitives for operations on collections and incrementalize selected example programs using those primitives, confirming expected asymptotic speedups.

show abstract

DBToaster: higher-order delta processing for dynamic, frequently fresh views

Cited by 73 publications

References 42 publications

Stale view cleaning

Stale view cleaning

Data Cleaning In Data Warehouse: A Survey of Data Pre-processing Techniques and Tools

Incremental $$\lambda $$ -Calculus in Cache-Transfer Style

Contact Info

Product

Resources

About