PDSPTF: Polystore Database System for Scalability and Access to PTF Time-Domain Astronomy Data Archives

Shrestha, Shashank; Poudel, Manoj; Wu, Yingjie; Chu, Wanming; Bhalla, Subhash; Kupfer, Thomas; Kulkarni, S. R.

doi:10.1007/978-3-030-14177-6_7

Cited by 3 publications

(2 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These telescopes observe the night sky in the visible and infrared spectra. It is a fully automated, wide-field survey aimed at the systematic exploration of the optical transient sky [35]. Two automatic reduction pipelines received data from the camera.…”

Section: Science Project and Virtual Observatory In The Era Of Big Da...mentioning

confidence: 99%

A Survey of Big Data Archives in Time-Domain Astronomy

et al. 2022

Self Cite

View full text Add to dashboard Cite

The rise of big data has resulted in the proliferation of numerous heterogeneous data stores. Even though multiple models are used for integrating these data, combining such huge amounts of data into a single model remains challenging. There is a need in the database management archives to manage such huge volumes of data without any particular structure which comes from unconnected and unrelated sources. These data are growing in size and thus demand special attention. The speed with which these data are growing as well as the varied data types involved and stored in scientific archives is posing further challenges. Astronomy is also increasingly becoming a science which is now based on a lot of data processing and involves assorted data. These data are now stored in domain-specific archives. Many astronomical studies are producing large-scale archives of data and these archives are then published in the form of data repositories. These mainly consist of images and text without any structure in addition to data with some structure such as relations with key values. When the archives are published as remote data repositories, it is challenging work to organize the data against their increased diversity and to meet the information demands of users. To address this problem, polystore systems present a new model of data integration and have been proposed to access unrelated data repositories using an uniform single query language. This article highlights the polystore system for integrating large-scale heterogeneous data in the astronomy domain.

show abstract

Section: Science Project and Virtual Observatory In The Era Of Big Da...mentioning

confidence: 99%

A Survey of Big Data Archives in Time-Domain Astronomy

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…In contrast, the other current solutions available for querying PTF data require the user to write complex programs in order to perform such complex queries. A comparison of the current system and the IRSA web system is presented in this research [44], where various queries from the time-domain astronomy are analyzed.…”

Section: Existing Ipac Sourcesmentioning

confidence: 99%

Processing Analytical Queries over Polystore System for a Large Astronomy Data Repository

et al. 2022

Self Cite

View full text Add to dashboard Cite

There are extremely large heterogeneous databases in the astronomical data domain, which keep increasing in size. The data types vary from images of astronomical objects to unstructured texts, relations, and key-values. Many astronomical data repositories manage such kinds of data. The Zwicky Transient Facility (ZTF) is one such data repository with a large amount of data with different varieties. Handling different types of data in a single database may have performance and efficiency issues. In this study, we propose a web-based query system built around the Polystore database architecture, and attempt to provide a solution for the growing size of data in the astronomical domain. The proposed system will unify querying over multiple datasets directly, thereby eliminating the effort to translate complex queries and simplify the work for the users in the astronomical domain. In this proposal, we study the models of data integration, analyze them, and incorporate them into a system to manage linked open data provided by astronomical domain. The proposed system is scalable, and its model can be used for various other systems to efficiently manage heterogeneous data.

show abstract