Compression-Aware Algorithms for Massive Datasets

Abstract

While massive datasets are often stored in compressed format, most algorithms are designed to operate on uncompressed data. We address this growing disconnect by developing a framework for compression-aware algorithms that operate directly on compressed datasets. Synergistically, we also propose new algorithmicallyaware compression schemes that enable algorithms to efficiently process compressed data. In particular, we apply this general methodology to geometric / CAD datasets that are ubiquitous in areas such as graphics, VLSI, and geographic information systems. We develop algorithms and corresponding compression schemes that address different types of datasets, including pointsets and graphs. Our methods are more efficient than their classical counterparts, and they extend to both lossless and lossy compression scenarios. This motivates further investigation of how this approach can enable algorithms to process ever-increasing big data volumes.

RSS Fingerprinting Dataset Size Reduction Using Feature-Wise Adaptive k-Means Clustering

“…Given that computing efficiency, dimensionality reduction, and data compression are highly demanded in IPS, different authors have proposed multiple methods based on clustering [4], radio-map reduction [5], [6] and other complex-search algorithms [7]. These methods may be executed in dedicated servers, smartphones, and even in low profile devices.…”