Probabilistic Record Linkage in Astronomy: Directional Cross-Identification and Beyond

Abstract

Modern astronomy increasingly relies upon systematic surveys, whose dedicated telescopes continuously observe the sky across varied wavelength ranges of the electromagnetic spectrum; some surveys also observe non-electromagnetic "messengers," such as high-energy particles or gravitational waves. Stars and galaxies look different through the eyes of different instruments, and their independent measurements have to be carefully combined to provide a complete, sound picture of the multicolor and eventful universe. The association of an object's independent detections is, however, a difficult problem scientifically, computationally, and statistically, raising varied challenges across diverse astronomical applications. The fundamental problem is finding records in survey databases with directions that match to within the direction uncertainties. Such astronomical versions of the record linkage problem are known by various terms in astronomy: cross-matching, cross-identification, and directional, positional, or spatio-temporal coincidence assessment. Astronomers have developed several statistical approaches for such problems, largely independently of related developments in other disciplines. Here we review emerging approaches that compute (Bayesian) probabilities for the hypotheses of interest: possible associations, or demographic properties of a cosmic population that depend on identifying associations. Many cross-identification tasks can be formulated within a hierarchical Bayesian partition model framework, with components that explicitly account for astrophysical effects (e.g., source brightness vs. wavelength, source motion, or source extent), selection effects, and measurement error. We survey recent developments, and highlight important open areas for future research.1 These images were produced by Konrad Dennerl and Wolfgang Voges for a 1994 conference poster; see Dennerl et al. (1994) and http://www.mpe-garching.mpg.de/background-picture.html for details.

Creating longitudinal datasets and cleaning existing data identifiers in a cystic fibrosis registry using a novel Bayesian probabilistic approach from astronomy

“…A Bayesian approach that utilises prior information, has been developed in Astronomy to solve the problem of cross matching galaxy catalogues by [ 13 , 25 ]. Their general probabilistic formalism for cross-identifying astronomical point sources, allow the folding in of expert knowledge on the physics of objects in a hierarchical framework, to help in the cross-identification process.…”

Towards Matching User Mobility Traces in Large-Scale Datasets

“…Their approach and motivation is in many ways similar to previous work on statistical databases [23,24], with considering more general cases. Further work focused on linking Bitcoin addresses to IP addresses, showing the limits of privacy [31,32], while the problem of record linkage has important applications in disciplines like biology or astronomy, facilitating probabilistic matching of data obtained from different measurements [33,34].…”

Astronomical observations: a guide for allied researchers

“…To facilitate this analysis, astronomers are beginning to embrace sophisticated database methods, including the use of Structured Query Language (SQL), spurred on by the advent of the Sloan Digital Sky Survey and other large projects. As in many other scientific fields, astronomers are rapidly expanding their use of machine learning, with topics such as classification [17,18] and cross-identification [10] being of particular interest. Real-time classification of time-variable events will be especially important for upcoming facilities such as LSST [e.g.…”