The Snowflake Elastic Data Warehouse

Dageville, Benoît; Cruanes, Thierry; Żukowski, Marcin; Антонов, В. А.; Avanes, Artin; Bock, Jonathan A.; Claybaugh, Jonathan; Engovatov, Daniel; Hentschel, Martin; Huang, Jiansheng; Lee, Allison W.; Motivala, Ashish; Munir, Abdul Q.; Pelley, Steven; Povinec, Peter; Rahn, Greg; Triantafyllis, Spyridon; Unterbrunner, Philipp

doi:10.1145/2882903.2903741

Cited by 213 publications

(82 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For example, authors used it to recommend projections in the Vertica column-store system. Snowflake [65] is an another commercial system capable of column-oriented data processing for semi-structured data. It adopts a Software-as-a-Service model and aims to free user from complex management tasks.…”

Section: Database Tuning For Column-storesmentioning

confidence: 99%

The design of an adaptive column-store system

Chernishev¹

2017

J Big Data

View full text Add to dashboard Cite

Section: Database Tuning For Column-storesmentioning

confidence: 99%

The design of an adaptive column-store system

Chernishev¹

2017

J Big Data

View full text Add to dashboard Cite

“…Nowadays, most analytics systems have adopted data skipping. Examples include Amazon Redshift [8], Google Powerdrill [1], Hive [43], IBM DB2 [40], Parquet [2], Vertica [11], Snowflake [16] and so on. Data skipping can be used to reduce data scan whether the underlying data layout is row-major or column-major.…”

Section: Related Workmentioning

confidence: 99%

“…Data skipping has become an essential mechanism for improving query performance in modern analytics databases (e.g., [1,8,16,26,40]) and the Hadoop ecosystem (e.g., [2,43]). In these systems, data are organized into blocks, each of which typically contains tens of thousands of tuples.…”

Section: Introductionmentioning

confidence: 99%

Skipping-oriented partitioning for columnar layouts

et al. 2016

View full text Add to dashboard Cite

As data volumes continue to grow, modern database systems increasingly rely on data skipping mechanisms to improve performance by avoiding access to irrelevant data. Recent work [39] proposed a fine-grained partitioning scheme that was shown to improve the opportunities for data skipping in row-oriented systems. Modern analytics and big data systems increasingly adopt columnar storage schemes, and in such systems, a row-based approach misses important opportunities for further improving data skipping. The flexibility of column-oriented organizations, however, comes with the additional cost of tuple reconstruction. In this paper, we develop Generalized Skipping-Oriented Partitioning (GSOP), a novel hybrid data skipping framework that takes into account these rowbased and column-based tradeoffs. In contrast to previous columnoriented physical design work, GSOP considers the tradeoffs between horizontal data skipping and vertical partitioning jointly. Our experiments using two public benchmarks and a real-world workload show that GSOP can significantly reduce the amount of data scanned and improve end-to-end query response times over the state-of-the-art techniques.

show abstract

“…Relational data systems have been increasingly adopting LSM-style of updates. My-Rocks [28] uses RocksDB as storage engine and SQLite4 [62] has experimented with LSM-trees in its storage layer, while columnar systems use LSM-style updates [20,44,63,70].…”

Section: Introductionmentioning

confidence: 99%

Lethe: A Tunable Delete-Aware LSM Engine

Sarkar

Papon

Staratzis

et al. 2020

Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

Data-intensive applications fueled the evolution of log structured merge (LSM) based key-value engines that employ the out-of-place paradigm to support high ingestion rates with low read/write interference. These benefits, however, come at the cost of treating deletes as a second-class citizen. A delete inserts a tombstone that invalidates older instances of the deleted key. State-of-the-art LSM engines do not provide guarantees as to how fast a tombstone will propagate to persist the deletion. Further, LSM engines only support deletion on the sort key. To delete on another attribute (e.g., timestamp), the entire tree is read and re-written. We highlight that fast persistent deletion without affecting read performance is key to support: (i) streaming systems operating on a window of data, (ii) privacy with latency guarantees on the right-to-be-forgotten, and (iii) en masse cloud deployment of data systems that makes storage a precious resource.To address these challenges, in this paper, we build a new key-value storage engine, Lethe, that uses a very small amount of additional metadata, a set of new delete-aware compaction policies, and a new physical data layout that weaves the sort and the delete key order. We show that Lethe supports any user-defined threshold for the delete persistence latency offering higher read throughput (1.17 − 1.4×) and lower space amplification (2.1 − 9.8×), with a modest increase in write amplification (between 4% and 25%). In addition, Lethe supports efficient range deletes on a secondary delete key by dropping entire data pages without sacrificing read performance nor employing a costly full tree merge.

show abstract

The Snowflake Elastic Data Warehouse

Cited by 213 publications

References 21 publications

The design of an adaptive column-store system

The design of an adaptive column-store system

Skipping-oriented partitioning for columnar layouts

Lethe: A Tunable Delete-Aware LSM Engine

Contact Info

Product

Resources

About