PQk-means

Abstract

Data clustering is a fundamental operation in data analysis. For handling large-scale data, the standard k-means clustering method is not only slow, but also memory-ine cient. We propose an efcient clustering method for billion-scale feature vectors, called PQk-means. By rst compressing input vectors into short productquantized (PQ) codes, PQk-means achieves fast and memory-e cient clustering, even for high-dimensional vectors. Similar to k-means, PQk-means repeats the assignment and update steps, both of which can be performed in the PQ-code domain. Experimental results show that even short-length (32 bit) PQ-codes can produce competitive results compared with k-means. is result is of practical importance for clustering in memory-restricted environments. Using the proposed PQk-means scheme, the clustering of one billion 128D SIFT features with K = 10 5 is achieved within 14 hours, using just 32 GB of memory consumption on a single computer.

Large-Scale Stream K-means Based on Product-Quantized Codes

“…• PQK-means [36]: Based on symmetric distance, PQK-means assigns each sample to the nearest cluster center. It then iteratively updates the cluster centers and sample labels.…”

“…Similarly, Shen et al [35] introduced auxiliary binary representation and designed an objective function based on Hamming distance. Matsui et al [36] compressed input vectors into short product quantization codes, achieving fast and efficient clustering. These methods significantly reduce data storage space through quantization, reduce clustering computation and processing time, and provide effective solutions for handling large-scale datasets.…”

Navigation pattern extraction from AIS trajectory big data via topic model

“…More precisely, an intermediate code is introduced to compress the original vector dataset to a lower dimension and short-length dataset. Based on this concept, a billion-scale memory-efficient clustering algorithm has been developed, called the PQk-means algorithm (Matsui et al, 2017).…”

“…where 𝑁 𝑘 = 𝑁 𝑖=1 𝐼 (𝐶 (𝑖) = 𝑘) is the number of vectors of the 𝑘th cluster, and 𝑑 (c 𝑖 , c 𝑘 is the distance between the 𝑖th code vector and 𝑘th code vector. With this algorithm, clustering a one billion dataset with 100,000 clusters is achieved in 14 h on a 32GB RAM personal computer (Matsui et al, 2017). Unlike other existing large-scale clustering methods, such as Bk-means (Gong et al, 2015) and IQkmeans (Avrithis et al, 2015), the original vector sample can also be reconstructed approximately from a given code, this being particularly efficient for large AIS trajectory data.…”

“…However, the bottleneck issue we have to deal with is the large amount of incoming AIS data. As it has been shown that the k-means vector quantisation approach does not perform well in the setting of large datasets, we use a high-performance PQk-means algorithm recently introduced (Matsui et al, 2017). Overall, a combination of topic model and extended vector quantisation is applied to then extract specific navigation patterns for a given maritime region of interest.…”