Data Discovery and Anomaly Detection Using Atypicality: Theory

Abstract

A central question in the era of 'big data' is what to do with the enormous amount of information. One possibility is to characterize it through statistics, e.g., averages, or classify it using machine learning, in order to understand the general structure of the overall data. The perspective in this paper is the opposite, namely that most of the value in the information in some applications is in the parts that deviate from the average, that are unusual, atypical. We define what we mean by 'atypical' in an axiomatic way as data that can be encoded with fewer bits in itself rather than using the code for the typical data. We show that this definition has good theoretical properties. We then develop an implementation based on universal source coding, and apply this to a number of real world data sets.

Data Discovery and Anomaly Detection Using Atypicality for Real-Valued Data

“…The issue is that there are an astonishing large number of such trees; for example for there are 676 such trees. Instead of choosing the best, we can use the idea of the CTW [ 1 , 54 , 55 ] and weigh in each node: Suppose after passing a signal of an internal node S through low-pass and high-pass filters and downsampler, and are produced in the children nodes of S . The weighted probability of in the internal node S will be which is a good coding distribution for both a memoryless source and a source with memory [ 54 , 55 ].…”

“…We therefore need universal methods. In the paper [ 1 ] we developed a methodology, atypicality, that can be used to discover such data. The basic idea is that if some data can be encoded with a shorter codelength in itself, i.e., with a universal source coder, rather than using the optimum coder for typical data, then it is atypical.…”

“…In our paper [ 1 ] we used universal source coding for anomaly detection; in [ 3 , 4 , 5 ] the authors used a type of universal empirical histogram. This kind of methodology is feasible when data is discrete.…”

“…It also encodes the length of the atypical subsequence using Rissanen’s universal coder for the integers [ 14 ], adding to the code length, so that the decoder knows when to switch back to the typical coder. The whole sequence is sequentially decodable, thus has a valid probability, and we know from [ 1 ] that this gives a valid criterion, at least for iid sequences, in the sense that not the whole sequence will be classified as atypical; the key is the insistence on decodability. It would be difficult to do this directly using Bayesian analysis, as we would have to develop probability distributions for the total sequence for every combination of atypical subsequences in the long sequence.…”

Coding of Graphs with Application to Graph Anomaly Detection

“…The papers [19], [20] show that atypicality has many desirable theoretical properties and that it works experimentally for sequences. Specifically for anomaly detection, the paper [20] shows that atypicality is (asymptotically) optimum for finite state machine (FSM) We will say that two FSM are distinct if they have no identical classes.…”