Finding the Number of Clusters in Unlabeled Datasets using Extended Dark Block Extraction

Abstract

Clustering analysis is the problem of partitioning a set of objects O = {o1… on} into c self-similar subsets based on available data. In general, clustering of unlabeled data poses three major problems: 1) assessing cluster tendency, i.e., how many clusters to seek? 2) Partitioning the data into c meaningful groups, and 3) validating the c clusters that are discovered. We address the first problem, i.e., determining the number of clusters c prior to clustering. Many clustering algorithms require number of clusters as an input parameter, so the quality of the clusters mainly depends on this value. Most methods are post clustering measures of cluster validity i.e., they attempt to choose the best partition from a set of alternative partitions.In contrast, tendency assessment attempts to estimate c before clustering occurs. Here, we represent the structure of the unlabeled data sets as a Reordered Dissimilarity Image (RDI), where pair wise dissimilarity information about a data set including ‗n' objects is represented as nxn image. RDI is generated using VAT (Visual Assessment of Cluster tendency), RDI highlights potential clusters as a set of -dark blocks‖ along the diagonal of the image. So, number of clusters can be easily estimated using the number of dark blocks across the diagonal. We develop a new method called -Extended Dark Block Extraction (EDBE) for counting the number of clusters formed along the diagonal of the RDI. EDBE method combines several image and signal processing techniques.

An adaptive method to determine the number of clusters in clustering process

“…Asadi et al [6] represent the structure of unlabeled data sets as a RDI (Reordered Dissimilarity Image). RDI highlights potential clusters as a set of "dark block" along the diagonal of image.…”

Cross layer design for RFID systems

“…However, they require the output of a clustering process while SSDA could run without a related clustering process. Some other schemes, like DBSCAN [19] or VAT-based schemes [4,46,61] can operate independently as well, but they incur large computational overhead, i.e., O(l 2 ). On the other hand, SSDA requires only O(l) overhead to achieve a similar accuracy level.…”

“…In Figure 3.11(b), we compare the accuracy between EA-k model and the SSDA algorithm in [28], which can estimate up to 4 colliding tags. We implement SSDA using the original settings in [28] and set the k range to [1,4] for fair comparison. We observe from Figure 3.11(b) that SSDA achieves higher accuracy (≈ 90%) than our EA-k model does.…”