Processing Analytical Queries over Polystore System for a Large Astronomy Data Repository

Abstract

There are extremely large heterogeneous databases in the astronomical data domain, which keep increasing in size. The data types vary from images of astronomical objects to unstructured texts, relations, and key-values. Many astronomical data repositories manage such kinds of data. The Zwicky Transient Facility (ZTF) is one such data repository with a large amount of data with different varieties. Handling different types of data in a single database may have performance and efficiency issues. In this study, we propose a web-based query system built around the Polystore database architecture, and attempt to provide a solution for the growing size of data in the astronomical domain. The proposed system will unify querying over multiple datasets directly, thereby eliminating the effort to translate complex queries and simplify the work for the users in the astronomical domain. In this proposal, we study the models of data integration, analyze them, and incorporate them into a system to manage linked open data provided by astronomical domain. The proposed system is scalable, and its model can be used for various other systems to efficiently manage heterogeneous data.

Enhancing Data Management Strategies with a Hybrid Layering Framework in Assessing Data Validation and High Availability Sustainability

“…With data migration skill enhancement, peopleware requires collaboration between multiple teams, including data administrators, developers, IT operations, and business stakeholders. A challenging aspect of data migration is the heterogeneity of suitable methods for data compatibility, accuracy, reliability, transparency, transition preparation, and persistent operations [19,53]. Multiple considerations must be optimized for data migration concerns.…”