Response to Comment on "Clustering by Passing Messages Between Data Points"

Abstract

Affinity propagation (AP) can be viewed as a generalization of the vertex substitution heuristic (VSH), whereby probabilistic exemplar substitutions are performed concurrently. Although results on small data sets (≤900 points) demonstrate that VSH is competitive with AP, we found VSH to be prohibitively slow for moderate-to-large problems, whereas AP was much faster and could achieve lower error.A ffinity propagation (AP) is an algorithm that clusters data and identifies exemplar data points that can be used for summarization and subsequent analysis (1). Dozens of clustering algorithms have been invented in the past 40 years, but in (1) we compared AP with three commonly used methods and found that AP could find solutions with lower error and do so much more quickly. Brusco and Köhn (2) compared AP with the best of 20 runs of a randomly initialized vertex substitution heuristic (VSH) described in 1997 (3), which is based on a previously introduced method (4). They found that for some small data sets (≤900 data points), VSH achieves lower error than AP in a similar amount of time. We subsequently confirmed those results but found no factual errors in our original report. Interestingly, when we studied larger, more complex data sets, we found that AP can achieve lower error than VSH in a fraction of the amount of time (5). VSH took ∼10 days to find 454 clusters in 17,770 Netflix movies, whereas AP took ∼2 hours and achieved lower error.As explained in our original report (1), regardless of whether or not the measure of data similarity is symmetric, "[e]xactly minimizing [AP's cost function] is computationally intractable, because a special case of this minimization problem is the NP-hard k-median problem" (also known as the p-median model, or PMM).[Also see (6).] Consequently, it is expected that different algorithms may work better for different data sets. In fact, we reported results using a brute force method that can exactly solve trivially small problems (e.g., 70 points and six clusters) in a minute or two on any modern computer. We also pointed out that linear programming relaxations (i.e., Lagrangian relaxations) have been used when the data set is not too large (7).We were curious about how close AP and VSH could get to the best possible (exact) solution, so we studied the original Olivetti face data set (n = 400 data points) for which the exact clustering solution could be found. Because the error for VSH varies depending on the random initialization, in all of our experiments we used the best of 20 runs. Figure 1A plots squared error versus the number of exemplars (k) for AP (single run), VSH (best of 20 runs), k-centers clustering (all of one million runs), and the exact solution (7). Both AP and VSH performed substantially better than k-centers clustering and, for practical purposes, achieved the exact solution (for k < 135, the difference in error between AP or VSH and the exact solution is less than the error reduction obtained by including an additional exemplar) (5).The results presented by Brusco...

Text Clustering with Seeds Affinity Propagation

“…The most significant advantage of SAP is that it is better than k-means in the foregoing evaluations while k-means runs 200 times (the best run is used to compare with SAP) and costs about twenty-fold of SAP in time. This result also confirms the one in [7], [30], and [31]. Furthermore, even after 10000 runs of k-means -with a size of 400 documents (F-measure: 0.406; Entropy: 0.677), we can't get similar results as SAP.…”

“…Many detailed analysis of the AP approach have been carried out (see for instance [30] and [31]) for various datasets with different scales. These studies show that for small datasets, there are only minor differences between traditional strategies (such as p-median model and vertex substitution heuristic) and Affinity Propagation clustering for both precision and CPU execution time.…”

“…These studies show that for small datasets, there are only minor differences between traditional strategies (such as p-median model and vertex substitution heuristic) and Affinity Propagation clustering for both precision and CPU execution time. Nevertheless, for large datasets, AP offers obvious advantages over existing methods [7,31]. In particular, in their work Frey and Dueck showed that an improvement in execution time of roughly 100 times is achieved on datasets of more than 10000 objects and ca.…”

“…500 clusters. Moreover, in [7][8][9][10][11][12] and [30][31][32][33], it has been identified that the similarity measurement has a great influence on AP clustering.…”

Affinity Propagation Clustering Using Path Based Similarity

“…To combat the second challenge, Frey et al [8] proposed a clustering technique called affinity propagation clustering (APC), which propagates affinity message between samples to search a high-quality set of clusters [9]. APC has been shown its usefulness in image segmentation [10,11], gene expressions [12] and text summarization [13].…”