A distributed tree data structure for real-time OLAP on cloud architectures

Abstract

Abstract-In contrast to queries for on-line transaction processing (OLTP) systems that typically access only a small portion of a database, OLAP queries may need to aggregate large portions of a database which often leads to performance issues. In this paper we introduce CR-OLAP, a Cloud based Real-time OLAP system based on a new distributed index structure for OLAP, the distributed PDCR tree, that utilizes a cloud infrastructure consisting of (m + 1) multi-core processors. With increasing database size, CR-OLAP dynamically increases m to maintain performance. Our distributed PDCR tree data structure supports multiple dimension hierarchies and efficient query processing on the elaborate dimension hierarchies which are so central to OLAP systems. It is particularly efficient for complex OLAP queries that need to aggregate large portions of the data warehouse, such as "report the total sales in all stores located in California and New York during the months February-May of all years". We evaluated CR-OLAP on the Amazon EC2 cloud, using the TPC-DS benchmark data set. The tests demonstrate that CR-OLAP scales well with increasing number of processors, even for complex queries. For example, on an Amazon EC2 cloud instance with eight processors, for a TPC-DS OLAP query stream on a data warehouse with 80 million tuples where every OLAP query aggregates more than 50% of the database, CR-OLAP achieved a query latency of 0.3 seconds which can be considered a real time response.

Large-Scale Indexing, Discovery, and Ranking for the Internet of Things (IoT)

“…We have provided more details about each one of these systems in the Appendix F. Interested reader can refer to the work of [Cugola and Margara 2012;Zámečníková and Kreslíková 2015] for a detailed discussion about di erent streaming systems. Overall, most of these approaches have an indexing structure that is based on one-dimensional feature and do not o er answers to the queries that might need aggregation or join for large portions of published data [Dehne et al 2013]. Although some of these systems support data gathered from real-world with a high rate, the solutions are not suitable for working with the uctuation of data rates in real-time [Kumbhare et al 2013].…”

“…Storm Trident 44 has been proposed to support stateful operators. However, Storm does not support multi-dimensional data which is crucial for IoT applications [Dehne et al 2013] in which the data is processed in the form of (key,value) pairs [Bahmani et al 2012].…”

An Effective Method to Answer OLAP Queries using R*-trees in Distributed Environment

“…A recent approach that has been reported in the literature for a cloud based real-time OLAP system uses the distributed PDCR tree [10]. It uses a cloud infrastructure consisting of multi-core processors.…”

Cloud Tree: A Library to Extend Cloud Services for Trees

“…* / 1 curN odeID ← 0; 2 if T he f irst key to insert then3 Create the root node as a leaf and store it into cloud. get tree node in cloud with curN odeID; 7 curV al ← get the data item in curN ode;8 if curV al == n then // Key already exists in the tree.…”