Parallel query processing in a polystore

Kranas, Pavlos; Kolev, Boyan; Levchenko, Oleksandra; Pacitti, Esther; Valduriez, Patrick; Jiménez-Peris, Ricardo; Patiño-Martı́nez, Marta

doi:10.1007/s10619-021-07322-5

Cited by 8 publications

(6 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Polystores also provide seamless access to cloud data stores. The CloudMdsQL Polystore provides a functional SQL-like query language to access many data sources (relational, NoSQL, and HDFS) [70].…”

Section: Challenge Of the Future Management In Astronomical Data Arch...mentioning

confidence: 99%

A Survey of Big Data Archives in Time-Domain Astronomy

et al. 2022

View full text Add to dashboard Cite

The rise of big data has resulted in the proliferation of numerous heterogeneous data stores. Even though multiple models are used for integrating these data, combining such huge amounts of data into a single model remains challenging. There is a need in the database management archives to manage such huge volumes of data without any particular structure which comes from unconnected and unrelated sources. These data are growing in size and thus demand special attention. The speed with which these data are growing as well as the varied data types involved and stored in scientific archives is posing further challenges. Astronomy is also increasingly becoming a science which is now based on a lot of data processing and involves assorted data. These data are now stored in domain-specific archives. Many astronomical studies are producing large-scale archives of data and these archives are then published in the form of data repositories. These mainly consist of images and text without any structure in addition to data with some structure such as relations with key values. When the archives are published as remote data repositories, it is challenging work to organize the data against their increased diversity and to meet the information demands of users. To address this problem, polystore systems present a new model of data integration and have been proposed to access unrelated data repositories using an uniform single query language. This article highlights the polystore system for integrating large-scale heterogeneous data in the astronomy domain.

show abstract

Section: Challenge Of the Future Management In Astronomical Data Arch...mentioning

confidence: 99%

A Survey of Big Data Archives in Time-Domain Astronomy

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Some of the key features of Apache Drill are (i) Low latency queries, which means that a simple query returns the result in a few milliseconds; (ii) Ability to access multiple data sources in a single query, such as Hive tables, JSON files and file systems (local or distributed); and (iii) it works with Business Intelligence tools, which allows for direct integration with specialized visualization tools [Drill 2022]. Apache Drill is considered a Polystore system [Kranas et al 2021] since it can query multiple DBMSs that follow multiple data models.…”

Section: Apachementioning

confidence: 99%

“…Although the aforementioned data management solutions (i.e., PostGIS and Nano Cubes) have been successfully used in many Vis projects, the Database community has been developing a series of domain/problem agnostic solutions for data management in recent years that can be applied in the Vis context, especially in interactive Vis. Such solutions range from NoSQL DBMSs (e.g., MonetDB [Boncz et al 2006], MongoDB [Makris et al 2021]) to Polystore systems [Kranas et al 2021] and large-scale data analysis frameworks such as Apache Spark (with Spark SQL) [Zaharia et al 2016], in addition to well-known RDBMSs like PostgreSQL. Although the aforementioned solutions are not designed to support interactive Vis systems, they have been shown to be increasingly efficient and can be used in many contexts, replacing domain-specific solutions such as Nano Cubes.…”

Section: Introductionmentioning

confidence: 99%

Beyond Click-and-View: a Comparative Study of Data Management Approaches for Interactive Visualization

Nascimento

Chagas

Lage

et al. 2022

jidm

View full text Add to dashboard Cite

Visual analytics solutions have been growing in popularity in recent years, not only for showing final results but also for assisting in interactive analysis and decision-making. Analysis of a large amount of data requires flexible exploration and visualizations. However, queries that span geographical regions over time slices are expensive to compute, which turns it challenging to accomplish interactive speeds for huge data sets. Such systems require efficient data availability, so that response time does not interfere with the user’s ability to observe and analyze. Simultaneously, researches in the database domain have proposed solutions that can be used to support visualization systems. This article presents a comparative study of data management approaches to support interactive visualizations. The chosen data management solutions are (i) Apache Drill (a Polystore system), (ii) Apache Spark (a big data framework), (iii) Elasticsearch (a search engine), (iv) MonetDB (a column-oriented DBMS), and (v) PostgreSQL (a relational DBMS). To evaluate the performance of each solution, we selected a list of spatiotemporal queries among multiple queries submitted by users in a visual analytics system for rainfall data analysis named TEMPO. The results of this study show that Apache Spark and MonetDB present the best performance for the selected queries.

show abstract

“…5 Thus, in recent years, the database community has developed different solutions for large-scale data management, such as NoSQL DBMS, for example, MonetDB 6 and MongoDB, 7 and Polystore systems. 8 Nevertheless, the most successful data analytics solutions have been the big data frameworks, notably Apache Hadoop and its alternatives like Apache Spark. 9 Spark improves the performance of applications by automatically exploiting parallelism and accomplishing in-memory data movement, in contrast to Hadoop, whose intermediate operations write to the storage volume.…”

mentioning

confidence: 99%

Optimizing computational costs of Spark for SARS‐CoV‐2 sequences comparisons on a commercial cloud

Nunes

Melo

Tadonki

et al. 2023

Concurrency and Computation

View full text Add to dashboard Cite

Cloud computing is currently one of the prime choices in the computing infrastructure landscape. In addition to advantages such as the pay-per-use bill model and resource elasticity, there are technical benefits regarding heterogeneity and large-scale configuration. Alongside the classical need for performance, for example, time, space, and energy, there is an interest in the financial cost that might come from budget constraints. Based on scalability considerations and the pricing model of traditional public clouds, a reasonable optimization strategy output could be the most suitable configuration of virtual machines to run a specific workload. From the perspective of runtime and monetary cost optimizations, we provide the adaptation of a Hadoop applications execution cost model extracted from the literature aiming at Spark applications modeled with the MapReduce paradigm. We evaluate our optimizer model executing an improved version of the Diff Sequences Spark application to perform SARS-CoV-2 coronavirus pairwise sequence comparisons using the AWS EC2's virtual machine instances. The experimental results with our model outperformed 80% of the random resource selection scenarios. By only employing spot worker nodes exposed to revocation scenarios rather than on-demand workers, we obtained an average monetary cost reduction of 35.66% with a slight runtime increase of 3.36%.

show abstract

Parallel query processing in a polystore

Cited by 8 publications

References 24 publications

A Survey of Big Data Archives in Time-Domain Astronomy

A Survey of Big Data Archives in Time-Domain Astronomy

Beyond Click-and-View: a Comparative Study of Data Management Approaches for Interactive Visualization

Optimizing computational costs of Spark for SARS‐CoV‐2 sequences comparisons on a commercial cloud

Contact Info

Product

Resources

About