Benchmarking SQL on MapReduce systems using large astronomy databases

Abstract

International audienceIn the era of bigdata, with a massive set of digital information of unprecedented volumes being collected and/or produced in several application domains , it becomes more and more difficult to manage and query large data repositories. In the framework of the PetaSky project (http://com.isima.fr/Petasky), we focus on the problem of managing scientific data in the field of cosmology. The data we consider are those of the LSST project (http://www.lsst.org/). The overall size of the database that will be produced is expected to exceed 60 PB [28]. In order to evaluate the performances of existing SQL On MapReduce data management systems, we conducted extensive experiments by using data and queries from the area of cosmology. The goal of this work is to report on the ability of such systems to support large scale declarative queries. We mainly investigated the impact of data partitioning, indexing and compression on query execution performances

ASTROIDE: A Unified Astronomical Big Data Processing Engine over Spark

“…Approaches based on distributed frameworks. In [28], authors report the ability of existing MapReduce management systems (Hive, HadoopDB) to support large scale declarative queries in the area of cosmology. But the query cases used in this work differs from our context because they don not cover the spatial queries, which are typical in astronomical applications.…”

Efficient astronomical query processing using spark

“…Approaches based on distributed frameworks. In [13], the authors report the ability of existing MapReduce management systems (Hive, HadoopDB) to support large scale declarative queries in the area of cosmology. But the query cases used in this work differs from our context because they don not cover the spatial queries, which are typical in astronomical applications.…”