Dolphin

Abstract

In this work a novel distance-based outlier detection algorithm, named DOLPHIN, working on disk-resident datasets and whose I/O cost corresponds to the cost of sequentially reading the input dataset file twice, is presented.It is both theoretically and empirically shown that the main memory usage of DOLPHIN amounts to a small fraction of the dataset and that DOLPHIN has linear time performance with respect to the dataset size. DOLPHIN gains efficiency by naturally merging together in a unified schema three strategies, namely the selection policy of objects to be maintained in main memory, usage of pruning rules, and similarity search techniques. Importantly, similarity search is accomplished by the algorithm without the need of preliminarily indexing the whole dataset, as other methods do.The algorithm is simple to implement and it can be used with any type of data, belonging to either metric or nonmetric spaces. Moreover, a modification to the basic method allows DOLPHIN to deal with the scenario in which the available buffer of main memory is smaller than its standard requirements. DOLPHIN has been compared with state-of-the-art distance-based outlier detection algorithms, showing that it is much more efficient.

Hierarchical Density Estimates for Data Clustering, Visualization, and Outlier Detection

“…This approach has some fundamental limitations Kriegel et al 2011]: (i) the choice of the density threshold is critical; (ii) often, it is not possible to simultaneously detect clusters of varied densities by using a single, global density threshold; and (iii) a flat clustering solution alone cannot describe possible hierarchical relationships that may exist between nested clusters lying on different density levels. Nested clusters at varied levels of density can only be described by hierarchical density-based clustering methods, such as those in Wishart [1969], Wong and Lane [1983], Ankerst et al [1999], Sander et al [2003], Brecheisen et al [2004], Chehreghani et al [2008], Stuetzle and Nugent [2010], Sun et al [2010], , and , which are able to provide more elaborated descriptions of a dataset at different degrees of granularity and resolution.…”

“…The hierarchy and the cluster tree produced by the central module (HDBSCAN*) can be postprocessed for multiple tasks. For instance, they are particularly suitable for interactive data exploration, as they can be easily transformed and visualized in different ways, such as an OPTICS reachability plot [Ankerst et al 1999;Sander et al 2003], a silhouette-like plot ], a detailed dendrogram, or a compacted cluster tree. In addition, for applications that expect a nonhierarchical partition of the data, the clustering hierarchy can also be postprocessed so that a flat solution-as the best possible nonhierarchical representation of the data in some sense-can be extracted.…”

“…In this section, we provide an extended description and discussion of our hierarchical clustering method, HDBSCAN* , which can be seen as a conceptual and algorithmic improvement over OPTICS [Ankerst et al 1999]. …”

“…This is a classic smoothing factor in density estimates whose behavior is well understood, and methods that have an analogous parameter (e.g., Ankerst et al [1999], , Pei et al [2009], and Stuetzle and Nugent [2010]) are typically robust to it.…”

Anomaly Detection Based on Regularized Vector Auto Regression in Thermal Power Plant

“…For example, sensor In actual situation, the number of the sensor is very large, and the speed of the data acquisition is high, so it is must to develop a fast and efficient scheme. To handling with high dimensional data, some schemes proposed [7][8][9][10]. Most of these methods share the common view: dimensional reduction.…”