RCFile: A fast and space-efficient data placement structure in MapReduce-based warehouse systems

He, Yongqiang; Lee, Rubao; Huai, Yin; Zheng, Shu; Jain, Namit; Zhang, Xiaodong; Xu, Zhiwei

doi:10.1109/icde.2011.5767933

Cited by 220 publications

(112 citation statements)

References 15 publications

Supporting

Mentioning

112

Contrasting

Order By: Relevance

“…Additionally, there has been extensive research into the performance and compression benefits of column-oriented storage within databases as noted in the related work. For Hive, RCFile's primary benefit is due to the additional compression that can be applied, rather than any performance benefit in lazily materializing rows [22].…”

Section: Columnarmentioning

confidence: 99%

Shark: Fast Data Analysis Using Coarse-grained Distributed Memory

Engle¹

2013

View full text Add to dashboard Cite

Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Copyright © 2013, by the author(s).All rights reserved.Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission.Shark: Fast Data Analysis Using Coarse-grained Distributed Memory Clifford EngleAbstract Shark is a research data analysis system built on a novel coarse-grained distributed shared-memory abstraction. Shark marries query processing with deep data analysis, providing a unified system for easy data manipulation using SQL and pushing sophisticated analysis closer to data. It scales to thousands of nodes in a fault-tolerant manner. Shark can answer queries 40X faster than Apache Hive and run machine learning programs 25X faster than MapReduce programs in Apache Hadoop on large datasets. This is a complete overview of the development of Shark, including design decisions, performance details, and comparison with existing data warehousing solutions. It demonstrates some of Shark's distinguishing features including its in-memory columnar caching and its unified machine learning interface.

show abstract

Section: Columnarmentioning

confidence: 99%

Shark: Fast Data Analysis Using Coarse-grained Distributed Memory

Engle¹

2013

View full text Add to dashboard Cite

show abstract

“…It can use less space for storage to reduce the disk I/O requirements, and at the same time it has more advantages in terms of the network transmission. Similar to RCFile [13], in the spatio-temporal grid index, we will store the data of the same period of time in column and will store the data of the different periods of time in row. In the QaDTree index, we will write the MBC data of leaf nodes according to the column into data block.…”

Section: The Storage Structure Of Spatio-temporal Datamentioning

confidence: 99%

A Novel Spatio-Temporal Data Storage and Index Method for ARM-Based Hadoop Server

Han

Huang

Yang

et al. 2016

Cloud Computing and Security

View full text Add to dashboard Cite

Abstract. During the past decade, a vast number of GPS devices have produced massive amounts of data containing both time and spatial information. This poses a great challenge for traditional spatial databases. With the development of distributed cloud computing, many highperformance cloud platforms have been built, which can be used to process such spatio-temporal data. In this research, to store and process data in an effective and green way, we propose the following solutions: firstly, we build a Hadoop cloud computing platform using Cubieboards2, an ARM development board with A20 processors; secondly, we design two types of indexes for different types of spatio-temporal data at the HDFS level. We use a specific partitioning strategy to divide data in order to ensure load balancing and efficient range query. To improve the efficiency of disk utilisation and network transmission, we also optimise the storage structure. The experimental results show that our cloud platform is highly scalable, and the two types of indexes are effective for spatio-temporal data storage optimisation and they can help achieve high retrieval efficiency.

show abstract

“…With the help of this technique they could significantly improve the query execution of range selection, update and OLAP queries. Other solutions like the RCFile approach [11] adopt the idea of the PAX system, but optimize the storage for specific use-cases. In the case of RCFile the data placement is optimized for data warehouses that follow the map reduce scheme.…”

Section: Related Workmentioning

confidence: 99%

A storage advisor for hybrid-store databases

et al. 2012

View full text Add to dashboard Cite

With the SAP HANA database, SAP offers a high-performance in-memory hybrid-store database. Hybrid-store databases-that is, databases supporting row-and column-oriented data management-are getting more and more prominent. While the columnar management offers high-performance capabilities for analyzing large quantities of data, the row-oriented store can handle transactional point queries as well as inserts and updates more efficiently. To effectively take advantage of both stores at the same time the novel question whether to store the given data row-or columnoriented arises. We tackle this problem with a storage advisor tool that supports database administrators at this decision. Our proposed storage advisor recommends the optimal store based on data and query characteristics; its core is a cost model to estimate and compare query execution times for the different stores. Besides a per-table decision, our tool also considers to horizontally and vertically partition the data and manage the partitions on different stores. We evaluated the storage advisor for the use in the SAP HANA database; we show the recommendation quality as well as the benefit of having the data in the optimal store with respect to increased query performance.

show abstract

RCFile: A fast and space-efficient data placement structure in MapReduce-based warehouse systems

Cited by 220 publications

References 15 publications

Shark: Fast Data Analysis Using Coarse-grained Distributed Memory

Shark: Fast Data Analysis Using Coarse-grained Distributed Memory

A Novel Spatio-Temporal Data Storage and Index Method for ARM-Based Hadoop Server

A storage advisor for hybrid-store databases

Contact Info

Product

Resources

About