Explaining anomalies in groups with characterizing subspace rules

Abstract

Anomaly detection has numerous applications and has been studied vastly. We consider a complementary problem that has a much sparser literature: anomaly description. Interpretation of anomalies is crucial for practitioners for sense-making, troubleshooting, and planning actions. To this end, we present a new approach called x-PACS (for eXplaining Patterns of Anomalies with Characterizing Subspaces), which "reverse-engineers" the known anomalies by identifying (1) the groups (or patterns) that they form, and (2) the characterizing subspace and feature rules that separate each anomalous pattern from normal instances. Explaining anomalies in groups not only saves analyst time and gives insight into various types of anomalies, but also draws attention to potentially critical, repeating anomalies. In developing x-PACS, we first construct a desiderata for the anomaly description problem. From a descriptive data mining perspective, our method exhibits five desired properties in our desiderata. Namely, it can unearth anomalous patterns (i) of multiple different types, (ii) hidden in arbitrary subspaces of a high dimensional space, (iii) interpretable by human analysts, (iv) different from normal patterns of the data, and finally (v) succinct, providing a short data description. No existing work on anomaly description satisfies all of these properties simultaneously. Furthermore, x-PACS is highly parallelizable; it is linear on the number of data points and exponential on the (typically small) largest characterizing subspace size. The anomalous patterns that x-PACS finds constitute interpretable "signatures", and while it is not our primary goal, they can be used for anomaly detection. Through extensive experiments on real-world datasets, we show the effectiveness and superiority of x-PACS in anomaly explanation over various baselines, and demonstrate its competitive detection performance as compared to the state-of-the-art.

A Unifying Review of Deep and Shallow Anomaly Detection

“…This is especially relevant in safety-critical environments [522], [523]. Existing work on interpretable AD has considered finding subspaces of anomaly discriminative features [458], [524]- [528], deducing sequential feature explanations [459], using featurewise reconstruction errors [56], [190], employing fully convolutional architectures [337], and explaining anomalies via integrated gradients [38] or LRP [334], [460]. In relation to the vast body of literature though, research on interpretability and trustworthiness in AD has seen comparatively little attention.…”

Anomaly explanation: A review

“…With x-PACKS [65], a subspace clustering is first performed on the dataset containing anomalies and normal data points. After that step, hyper-rectangles containing the maximum number of anomalous data points and the minimum number of regular instances are obtained.…”

A survey on outlier explanations

“…However, when the outlying attributes of an alert do not fit any known attacks, analysts can use the information to either flag the alert as a false alarm or investigate it further to define a new type of attack. Furthermore, a collection of the unknown attacks can be grouped using XPACS [ 64 ] described in Sect. 7.2.4 .…”

“…By doing so, the analyst can examine and identify the root cause of each subset of outliers at once. Grouping outliers can be done using an algorithm such as the XPACS [ 64 ] algorithm described in Sect. 7.2.4 .…”

“…We now describe XPACS [ 64 ], whose goal is to cluster outliers in an arbitrary subspace. By finding the clusters, it also finds the outlying attributes for each cluster.…”