Content-Aware Partial Compression for Textual Big Data Analysis in Hadoop

Abstract

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Abstract-A substantial amount of information in companies and on the Internet is present in the form of text. The value of this semi-structured and unstructured data has been widely acknowledged, with consequent scientific and commercial exploitation. The ever-increasing data production, however, pushes data analytic platforms to their limit. Compression as an effective means to reduce data size has been employed by many emerging data analytic platforms, whom the main purpose of data compression is to save storage space and reduce data transmission cost over the network. Since general purpose compression methods endeavour to achieve higher compression ratios by leveraging data transformation techniques and contextual data, this context-dependency forces the access to the compressed data to be sequential. Processing such compressed data in parallel, such as desirable in a distributed environment, is extremely challenging. This work proposes techniques for more efficient textual big data analysis with an emphasis on content-aware compression schemes suitable for the Hadoop analytic platform. The compression schemes have been evaluated for a number of standard MapReduce analysis tasks using a collection of public and private real-world datasets. In comparison with existing solutions, they have shown substantial improvement in performance and significant reduction in system resource requirements.

A Comprehensive Framework for Big Data Analytics in Education

“…This work has presented a comprehensive representation scheme of the involuntary regulation of the behavior of student. Apart from this, there are various other schemes towards big data analytics e.g., tensor-based scheme [27], compression based on context [28], pattern analysis [29], deep learning [30], clustering technique [31]. Existing system has also witnessed an extensive usage of Hadoop framework.…”

SecHDFS: Efficient and Secure Data Storage Model over HDFS using RC6

“…The limitation of Fast-Sec approach is to encryption size is large for a given plain test size, which increases storage space. Record aware partial compression (RAPC) has proposed by Dong et al [7], which saves storage space . The plaintext data and security keys are not exposed to main memory and secondary storage.…”

Storage Space Allocation Strategy for Digital Data with Message Importance

“…This exceedingly massive data makes the conventional data storage mechanisms inadequate within a tolerable time, and therefore the data storage is one of the major challenges in big data [ 3 ]. Note that storing all the data becomes more and more dispensable nowadays, and it is also not conducive to reduce data transmission costs [ 4 , 5 ]. In fact, data compression storage is widely adopted in many applications, such as IoT [ 2 ], industrial data platform [ 6 ], bioinformatics [ 7 ], wireless networking [ 8 ].…”

“…For example, Reference [ 14 ] proposed an adaptive compression scheme in IoT systems, and Reference [ 15 ] investigated the backlog-adaptive source coding system in terms of age of information. In fact, most of the previous compression methods usually carried out compression by means of contextual data or leveraging data transformation techniques [ 4 ].…”