Density‐based clustering

Abstract

Clustering refers to the task of identifying groups or clusters in a data set. In density‐based clustering, a cluster is a set of data objects spread in the data space over a contiguous region of high density of objects. Density‐based clusters are separated from each other by contiguous regions of low density of objects. Data objects located in low‐density regions are typically considered noise or outliers. In this review article we discuss the statistical notion of density‐based clusters, classic algorithms for deriving a flat partitioning of density‐based clusters, methods for hierarchical density‐based clustering, and methods for semi‐supervised clustering. We conclude with some open challenges related to density‐based clustering. This article is categorized under: Technologies > Data Preprocessing Ensemble Methods > Structure Discovery Algorithmic Development > Hierarchies and Trees

Intruder Alert: Dimension Reduction and Density-Based Clustering for a Cybersecurity Application

“…Another problem that has already been dealt with in the literature is that noise points between clusters might be dense enough to chain together dense regions when they should be separated. The solution cited by Aggarwal [3] and by Campello [11], who attributes the idea to Wishart, is to eliminate non-dense points first. These are defined by Aggarwal as those having too few neighbours.…”

“…It is assumed that clusters are high density areas of an unknown probability density from which the dataset has been sample. Nearest neighbour and kernel estimators have often been used to find local density estimates [11].…”

“…Lower-and higher-density clusters can also be allowed by varying a threshold regionally. This can be particularly important because density levels may vary by data region, and some lower-density clusters could even have other higher-density clusters nested within them [11]. Hierarchical clustering has traditionally allowed these variations to be captured in the tree structure.…”

“…Campello and colleagues note that standard cluster validity measures rely on hyperspherical or globular clusters rather than arbitrary shapes to judge within-cluster compactness or variance and between-cluster separation, making them intractable in this type of problem [11]. They also note that most previous methods These core ideas may be summarized into the procedure used to calculate DBCV.…”

“…While streaming algorithms for some density-based clustering exist (e.g. [84]), in 2019 Campello et al listed this as an area for further efforts [11]. One might speculate that if time were treated as a local dimension then density-connectedness could be used to generate clusters in time.…”