Astronomaly: Personalised Active Anomaly Detection in Astronomical Data

Abstract

Survey telescopes such as the Vera C. Rubin Observatory and the Square Kilometre Array will discover billions of static and dynamic astronomical sources. Properly mined, these enormous datasets will likely be wellsprings of rare or unknown astrophysical phenomena. The challenge is that the datasets are so large that most data will never be seen by human eyes; currently the most robust instrument we have to detect relevant anomalies. Machine learning is a useful tool for anomaly detection in this regime. However, it struggles to distinguish between interesting anomalies and irrelevant data such as instrumental artefacts or rare astronomical sources that are simply not of interest to a particular scientist. Active learning combines the flexibility and intuition of the human brain with the raw processing power of machine learning. By strategically choosing specific objects for expert labelling, it minimises the amount of data that scientists have to look through while maximising potential scientific return. Here we introduce Astronomaly: a general anomaly detection framework with a novel active learning approach designed to provide personalised recommendations. Astronomaly can operate on most types of astronomical data, including images, light curves and spectra. We use the Galaxy Zoo dataset to demonstrate the effectiveness of Astronomaly, as well as simulated data to thoroughly test our new active learning approach. We find that for both datasets, Astronomaly roughly doubles the number of interesting anomalies found in the first 100 objects viewed by the user. Astronomaly is easily extendable to include new feature extraction techniques, anomaly detection algorithms and even different active learning approaches. The code is publicly available at https://github.com/MichelleLochner/astronomaly.

Real-time Detection of Anomalies in Multivariate Time Series of Astronomical Data

“…5], however, our approach may still flag unusual transient phenomena that don't align with our trained transient classes but are uninteresting to most astronomers. To deal with this, future work should consider Active Learning frameworks that use methods such as Human-in-the-loop learning that specifically target what users define as interesting phenomena (recent work by Ishida et al [8], Lochner & Bassett [15] have begun working on Active Learning for anomaly detection).…”

Preparing to discover the unknown with Rubin LSST -- I: Time domain

“…Notable advances in the field have been achieved in recent years across disciplines: from threat detection in defense and security (e.g. Sultani et al 2018), to astrophysics (Soraisam et al 2020;Pruzhinskaya et al 2019;Ishida et al 2019;Aleo et al 2020;Vafaei Sadr et al 2019;Martínez-Galarza et al 2020;Lochner & Bassett 2020;Doorenbos et al 2020) with the discovery of rare and possible unique astrophysical phenomena (Lintott et al 2009;Micheli et al 2018;Boyajian et al 2018, although we note that two of these "true novelties" were detected through crowd-sourced data analysis). Anomaly detection is generally approached either through unsupervised or supervised learning learning techniques (e.g.…”

“…2019; Ishida et al 2019;Aleo et al 2020;Vafaei Sadr et al 2019;Martínez-Galarza et al 2020;Lochner & Bassett 2020;Doorenbos et al 2020) we attempted to remain true to the premise that a true novelty is something that fundamentally cannot be predicted. This exercise is conceptually difficult as by definition we do not know what we are looking for.…”

A Census of Protostellar Outflows in Nearby Molecular Clouds

“…t-SNE is a tool to visualize high-dimensional data by converting the similarities between data points into a low-dimensional manifold. It has been used for classification and outlier detection in a variety of astronomy data (Jofré et al 2017;Reis et al 2018;Lochner & Bassett 2020;Fluke & Jacobs 2020). For example, Reis et al (2018) successfully applied t-SNE to cluster stars based on their spectra and identify outliers based on the t-SNE map.…”