Outlier Mining Methods Based on Graph Structure Analysis

Abstract

Outlier detection in high-dimensional datasets is a fundamental and challenging problem across disciplines that has also practical implications, as removing outliers from the training set improves the performance of machine learning algorithms. While many outlier mining algorithms have been proposed in the literature, they tend to be valid or efficient for specific types of datasets (time series, images, videos, etc.). Here we propose two methods that can be applied to generic datasets, as long as there is a meaningful measure of distance between pairs of elements of the dataset. Both methods start by defining a graph, where the nodes are the elements of the dataset, and the links have associated weights that are the distances between the nodes. Then, the first method assigns an outlier score based on the percolation (i.e., the fragmentation) of the graph. The second method uses the popular IsoMap non-linear dimensionality reduction algorithm, and assigns an outlier score by comparing the geodesic distances with the distances in the reduced space. We test these algorithms on real and synthetic datasets and show that they either outperform, or perform on par with other popular outlier detection methods. A main advantage of the percolation method is that is parameter free and therefore, it does not require any training; on the other hand, the IsoMap method has two integer number parameters, and when they are appropriately selected, the method performs similar to or better than all the other methods tested.

New algorithms to characterize and classify ophthalmic images

“…The second algorithm, described in Ref. 10 uses Isomap to learn the low-dimensional manifold in which the images are embedded, and assigns to each image an outlier score by comparing the geodesic distances with the distances in the low-dimensional space. Figure 2 shows the Image Map obtained from the two features returned by Isomap.…”

“…Figure 4 shows that the performance of the algorithm, measured by the correlation coefficient between the feature returned by the ordering algorithm and the feature provided by manual expert annotation (mean angle), improves when the images that contain artifacts (outliers detected by the outlier detection algorithms proposed in Ref. 10 ) are removed from the database. The different lines show results with different methods of outlier identification and the colored region indicates results when the images removed are randomly selected.…”

“…8 Here we first present novel algorithms to analyze a database of AS-OCT images: we demonstrate machine learning algorithms to order the images of an AS-OCT image database according to the degree of angle-closure, 9 and also, to identify the images in the database that contain artifacts. 10 In the second part we present novel algorithms for the analysis of color retinal fundus images. 11 These algorithms segment the images and return features of the retinal blood vessel network structure that quantify changes in patients diagnosed with glaucoma or with diabetic retinopathy.…”

Outlier mining in high-dimensional data using the Jensen–Shannon divergence and graph structure analysis

“…Many methods for outlier detection have been proposed in the literature (see, e.g. [2][3][4][5][6] and references therein), some of them, based on distances that can be computed between elements of the dataset [7][8][9][10][11]. In outlier detection via graph methods, distance-based outlier mining is based on a fully connected graph structure in which the nodes represent the elements of the dataset and the connections between them are quantified by a distance measure.…”

Editorial: Fundamentals and Applications of AI: An Interdisciplinary Perspective

“…The potential of graph-based methods in combination with machine learning algorithms is addressed in Ref. [5]. The authors explore two methods to detect outliers, with applications to highdimensional datasets.…”