The vertica analytic database

Lamb, Andrew; Fuller, Matt; Varadarajan, Ramakrishna; Tran, Nga; Vandiver, Ben; Doshi, Lyric; Bear, Chuck

doi:10.14778/2367502.2367518

Cited by 251 publications

(130 citation statements)

References 6 publications

Supporting

Mentioning

120

Contrasting

Unclassified

Order By: Relevance

“…All of these compression algorithms don't consider the characters of the data. In database management system area, e.g.,C-Store [9][10], HP Vertica [11], IBM DB2 BLU [12], a batch of compression techniques are employed to compress data based on data characteristics. These works are for column relational data management, but cannot used in graph data.…”

Section: Data Compressionmentioning

confidence: 99%

SimStore: Efficient Data Management for Network Propagation Simulation

Qu¹,

Zhang²,

Cao³

2016

Proceedings of the 2016 International Conference on Artificial Intelligence: Technologies and Applications

View full text Add to dashboard Cite

Abstract-Simulation is widely adopted in large scale network propagation analytics. Plenty of analytics scenarios require to retrieve, review the simulation status of a given time points or interval. Unfortunately, it is unaffordable to re-run the simulation due to the long running time and other costs in many cases. In this paper, we introduce a system (SimStore) to enable efficient storage and retrieval of simulation snapshots. We present a novel technique to compress a series of simulation snapshots in order to reduce the storage cost. Experimental results demonstrate the efficiency and effectiveness of the proposed methods.

show abstract

Section: Data Compressionmentioning

confidence: 99%

SimStore: Efficient Data Management for Network Propagation Simulation

Qu¹,

Zhang²,

Cao³

2016

Proceedings of the 2016 International Conference on Artificial Intelligence: Technologies and Applications

View full text Add to dashboard Cite

show abstract

“…Thus, Virtuoso does not have any concept of row number but rather has multi-part sorted column-wise compressed indices for all persistent data structures. In this, Virtuoso most resembles [1,7]. This structure is scale-out friendly since partitioning can be determined by any high cardinality key part and no global synthetic row number needs to exist, even as an abstraction.…”

Section: Virtuoso Column Storementioning

confidence: 99%

“…This is by and large not an OLTP workload but short transactions must still be efficient. Thus Virtuoso departs from the typical read-intensive optimization of column stores that have a separate delta structure periodically merged into a read-only columnwise compressed format [7,18]. Virtuoso updates column-wise compressed data in place, keeping locks positionally and associating rollback information to the lock when appropriate.…”

Section: Virtuoso Column Storementioning

confidence: 99%

Advances in Large-Scale RDF Data Management

Boncz

Erling

Pham

2014

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract.One of the prime goals of the LOD2 project is improving the performance and scalability of RDF storage solutions so that the increasing amount of Linked Open Data (LOD) can be efficiently managed. Virtuoso has been chosen as the basic RDF store for the LOD2 project, and during the project it has been significantly improved by incorporating advanced relational database techniques from MonetDB and Vectorwise, turning it into a compressed column store with vectored execution. This has reduced the performance gap ("RDF tax") between Virtuoso's SQL and SPARQL query performance in a way that still respects the "schema-last" nature of RDF. However, by lacking schema information, RDF database systems such as Virtuoso still cannot use advanced relational storage optimizations such as table partitioning or clustered indexes and have to execute SPARQL queries with many selfjoins to a triple table, which leads to more join effort than needed in SQL systems. In this chapter, we first discuss the new column store techniques applied to Virtuoso, the enhancements in its cluster parallel version, and show its performance using the popular BSBM benchmark at the unsurpassed scale of 150 billion triples. We finally describe ongoing work in deriving an "emergent" relational schema from RDF data, which can help to close the performance gap between relational-based and RDF-based storage solutions. General ObjectivesOne of the objectives of the LOD2 EU project is to boost the performance and the scalability of RDF storage solutions so that it can, efficiently manage huge datasets of Linked Open Data (LOD). However, it has been noted that given similar data management tasks, relational database technology significantly outperformed RDF data stores. One controlled scenario in which the two technologies can be compared is the BSBM benchmark [2], which exists equivalent relational and RDF variants. As illustrated in Fig. 1, while the SQL systems can process by up to 40-175K QMpH, the Triple stores can only reach 1-10K QMpH, showing a factor of 15-40 of performances difference.

show abstract

“…Consider the case of columnar database systems [16,8,10], where memory mapped storage could be used to represent in memory columnar data, backed on disk by a suitable file representation. During in memory data access the data is brought to the memory from disk and stays in memory as long as there is no need to swap it out.…”

Section: Introductionmentioning

confidence: 99%

NUMA obliviousness through memory mapping

Gawade

Kersten

2015

Proceedings of the 11th International Workshop on Data Management on New Hardware

View full text Add to dashboard Cite

With the rise of multi-socket multi-core CPUs a lot of effort is being put into how to best exploit their abundant CPU power. In a shared memory setting the multi-socket CPUs are equipped with their own memory module, and access memory modules across sockets in a non-uniform access pattern (NUMA). Memory access across socket is relatively expensive compared to memory access within a socket. One of the common solutions to minimize across socket memory access is to partition the data, such that the data affinity is maintained per socket.In this paper we explore the role of memory mapped storage to provide transparent data access in a NUMA environment, without the need of explicit data partitioning. We compare the performance of a database engine in a distributed setting in a multi-socket environment, with a database engine in a NUMA oblivious setting. We show that though the operating system tries to keep the data affinity to local sockets, a significant remote memory access still occurs, as the number of threads increase. Hence, setting explicit process and memory affinity results into a robust execution in NUMA oblivious plans. We use micro-experiments and SQL queries from the TPC-H benchmark to provide an in-depth experimental exploration of the landscape, in a four socket Intel machine.

show abstract

The vertica analytic database

Cited by 251 publications

References 6 publications

SimStore: Efficient Data Management for Network Propagation Simulation

SimStore: Efficient Data Management for Network Propagation Simulation

Advances in Large-Scale RDF Data Management

NUMA obliviousness through memory mapping

Contact Info

Product

Resources

About