HLC2: a highly efficient cross-matching framework for large astronomical catalogues on heterogeneous computing environments

Abstract

Cross-matching operation, which is to find corresponding data for the same celestial object or region from multiple catalogues, is indispensable to astronomical data analysis and research. Due to the large amount of astronomical catalogues generated by the ongoing and next generation large-scale sky surveys, the time complexity of the cross-matching is increasing dramatically. Heterogeneous computing environments provide a theoretical possibility to accelerate the cross-matching, but the performance advantages of heterogeneous computing resources have not been fully utilized. To meet the challenge of cross-matching for substantial increasing amount of astronomical observation data, this paper proposes HLC2 (Heterogeneous-computing-enabled Large Catalogue Cross-matcher), a high-performance cross-matching framework based on spherical position deviation on CPU-GPU heterogeneous computing platforms. It supports scalable and flexible cross-matching and can be directly applied to the fusion of large astronomical catalogues from survey missions and astronomical data centers. A performance estimation model is proposed to locate the performance bottlenecks and guide the optimizations. A two-level partitioning strategy is designed to generate an optimized data placement according to the positions of celestial objects to increase throughput. To make HLC2 a more adaptive solution, the architecture-aware task splitting, thread parallelization and concurrent scheduling strategies are designed and integrated. Moreover, a novel quad-direction strategy is proposed for the boundary problem to effectively balance performance and completeness. We have experimentally evaluated HLC2 using public released catalogue data. Experiments demonstrate that HLC2 scales well on different sizes of catalogues and the cross-matching speed is significantly improved compared to the state-of-the-art cross-matchers.

TT-HEALpix: A New Data Indexing Strategy for Efficient Cross-match of Large-scale Astronomical Catalogs

“…In the Formula (2), the use of a 3× angular distance is aim to capturing a greater number of candidate celestial bodies beyond the error range, preventing the omission of celestial bodies with larger positional inaccuracies. This is the universally applicable criterion for determining the success of a cross-matching, scholars such as Gao et al (2008a), Zhao et al (2010), Yu et al (2020), Zhang et al (2021), andZhang et al (2023) have been determined using this criterion.…”

“…The matching speed is 1.67 times faster than MySQL, but the accuracy is lower. On the other hand, Zhang et al (2023) proposed a highperformance cross-matching framework based on CPU-GPU heterogeneous computing to accelerate calculations and improve data throughput. However, the four-way strategy proposed to address the boundary problem does not fundamentally overcome the limitations of a single index structure.…”

A Cross-matching Service for Data Center of Xinjiang Astronomical Observatory

“…Li et al (2019) designed a multi-band catalog unified format, combined with the data layout strategy of minimum conflict to improve the parallelization of cross-matching, and achieved 30.3% and 30.7% time reduction compared with Quad Tree Cube (Q3C) and HealpiX-tree-C (H3C) at 200 million data sources of astronomical catalogs. Zhang et al (2023b) proposed a large-scale cross-matching framework supporting heterogeneous computing, which reduced the cross-matching time to 5 s for small-scale astronomical catalogs, 150 s for medium-scale astronomical catalogs, and 260 s for large-scale astronomical catalogs.…”

3DT-CM: A Low-complexity Cross-matching Algorithm for Large Astronomical Catalogues Using 3d-tree Approach

“…Index partitioning methods have been particularly useful, but they suffer from source leaking at the border of each area. To address this issue, mixed indexing (Yu et al 2020) and border data redundancy methods (Jia & Luo 2016) (Zhang et al 2023) have been introduced.…”

“…Hardware structure has also been utilized to speed up the cross-matching process, with various parallel computing methods being employed (Zečević et al 2019) (Zhang et al 2023). However, few of these methods have focused on improving the cross-matching algorithm itself.…”