Distributed architecture of Oracle database in-memory

Mukherjee, Nilanjana; Chavan, Shasank; Colgan, Maria; Das, Dinesh; Gleeson, Mike; Hase, Sanket; Holloway, Allison; Jin, Hui; Kamp, Jesse; Kulkarni, Kartik; Lahiri, Tirthankar; Loaiza, Juan; Macnaughton, Neil; Marwah, Vineet; Mullick, Atrayee; Witkowski, Andy; Yan, Jiaqi; Zaït, Mohamed

doi:10.14778/2824032.2824061

Cited by 21 publications

(12 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…4) [7] is the new inmemory area that hosts the IMCUs across the cluster. It is a shared-nothing container of IMCUs within an instance.…”

Section: Distributed Dbim -A Quick Reviewmentioning

confidence: 99%

“…• The transaction manager [7] ensures strict real-time consistency between the data blocks and their corresponding IMCUs. It also ensures that block modifications due to on-line transaction processing operations (OLTP) are propagated to the appropriate IMCUs across the cluster.…”

Section: Distributed Dbim -A Quick Reviewmentioning

confidence: 99%

“…• The distribution manager [7], as the name suggests, handles all the distribution, duplication, availability, and the fault tolerance aspects of the architecture. It also provides access awareness to the transaction manager and the SQL execution engine.…”

Section: Distributed Dbim -A Quick Reviewmentioning

confidence: 99%

“…The overall distributed architecture of Oracle Database in Memory and its scale-out aspects are highlighted in [5] [7]. Besides providing distribution and query execution scale out across a cluster of servers, the distributed architecture has been designed to provide real-time duplication of the inmemory columnar format for high availability along with fault tolerant query execution across server failures.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Fault-tolerant real-time analytics with distributed Oracle Database In-memory

Mukherjee

Chavan

Colgan

et al. 2016

2016 IEEE 32nd International Conference on Data Engineering (ICDE)

Self Cite

View full text Add to dashboard Cite

Modern data management systems are required to address new breeds of OLTAP applications. These applications demand real time analytical insights over massive data volumes not only on dedicated data warehouses but also on live mainstream production environments where data gets continuously ingested and modified. Oracle introduced the Database In-memory Option (DBIM) in 2014 as a unique dual row and column format architecture aimed to address the emerging space of mixed OLTAP applications along with traditional OLAP workloads. The architecture allows both the row format and the column format to be maintained simultaneously with strict transactional consistency. While the row format is persisted in underlying storage, the column format is maintained purely in-memory without incurring additional logging overheads in OLTP.Maintenance of columnar data purely in memory creates the need for distributed data management architectures. Performance of analytics incurs severe regressions in single server architectures during server failures as it takes non-trivial time to recover and rebuild terabytes of in-memory columnar format. A distributed and distribution aware architecture therefore becomes necessary to provide real time high availability of the columnar format for glitch-free in-memory analytic query execution across server failures and additions, besides providing scale out of capacity and compute to address real time throughput requirements over large volumes of inmemory data. In this paper, we will present the high availability aspects of the distributed architecture of Oracle DBIM that includes extremely scaled out application transparent column format duplication mechanism, distributed query execution on duplicated in-memory columnar format, and several scenarios of fault tolerant analytic query execution across the in-memory column format at various stages of redistribution of columnar data during cluster topology changes.

show abstract

“…4) [7] is the new inmemory area that hosts the IMCUs across the cluster. It is a shared-nothing container of IMCUs within an instance.…”

Section: Distributed Dbim -A Quick Reviewmentioning

confidence: 99%

Section: Distributed Dbim -A Quick Reviewmentioning

confidence: 99%

Section: Distributed Dbim -A Quick Reviewmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Fault-tolerant real-time analytics with distributed Oracle Database In-memory

Mukherjee

Chavan

Colgan

et al. 2016

2016 IEEE 32nd International Conference on Data Engineering (ICDE)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Oracle and SAP recently proposed approaches for hybrid distributed query and transaction processing, relying on a shared buffer cache [25] and a shared log [14], respectively. We plan to extend our work on hybrid processing [23] from full replication to fragmented relations to scale query performance while sustaining HyPer's excellent TX throughput.…”

Section: Network Bandwidthmentioning

confidence: 99%

High-speed query processing over high-speed networks

Rödiger¹,

Mühlbauer²,

Kemper³

et al. 2015

Proc. VLDB Endow.

View full text Add to dashboard Cite

Modern database clusters entail two levels of networks: connecting CPUs and NUMA regions inside a single server in the small and multiple servers in the large. The huge performance gap between these two types of networks used to slow down distributed query processing to such an extent that a cluster of machines actually performed worse than a single many-core server. The increased main-memory capacity of the cluster remained the sole benefit of such a scale-out.The economic viability of high-speed interconnects such as InfiniBand has narrowed this performance gap considerably. However, InfiniBand's higher network bandwidth alone does not improve query performance as expected when the distributed query engine is left unchanged. The scalability of distributed query processing is impaired by TCP overheads, switch contention due to uncoordinated communication, and load imbalances resulting from the inflexibility of the classic exchange operator model. This paper presents the blueprint for a distributed query engine that addresses these problems by considering both levels of networks holistically. It consists of two parts: First, hybrid parallelism that distinguishes local and distributed parallelism for better scalability in both the number of cores as well as servers. Second, a novel communication multiplexer tailored for analytical database workloads using remote direct memory access (RDMA) and low-latency network scheduling for high-speed communication with almost no CPU overhead. An extensive evaluation within the HyPer database system using the TPC-H benchmark shows that our holistic approach indeed enables high-speed query processing over high-speed networks.

show abstract