File Format s for Big Data Storage Systems

Abstract

Big data is one of the most influential technologies of the modern era. However, in order to support maturity of big data systems, development and sustenance of heterogeneous environments is requires. This, in turn, requires integration of technologies as well as concepts. Computing and storage are the two core components of any big data system. With that said, big data storage needs to communicate with the execution engine and other processing and visualization technologies to create a comprehensive solution. This brings the facet of big data file formats into picture. This paper classifies available big data file formats into five categories namely text-based, row-based, column-based, in-memory and data storage services. It also compares the advantages, shortcomings and possible use cases of available big data file formats for Hadoop, which is the foundation for most big data computing technologies. Lastly, it provides a discussion on tradeoffs that must be considered while choosing a file format for a big data system, providing a framework for creation for file format selection criteria.

Big Data Storage based on CAN Protocol

“…It is in this context, that the term Big Data appeared [2]. The term « big data » [3] refers to databases that are too large and complex to be studied with traditional statistical methods and, by extension, to all the new tools for analyzing these data [4]. This article focuses on two major issues associated with Big Data, first their storage and second their exploitation in a reasonable time.…”