Finding associations and computing similarity via biased pair sampling

Abstract

Abstract-Sampling-based methods have previously been proposed for the problem of finding interesting associations in data, even for low-support items. While these methods do not guarantee precise results, they can be vastly more efficient than approaches that rely on exact counting. However, for many similarity measures no such methods have been known. In this paper we show how a wide variety of measures can be supported by a simple biased sampling method. The method also extends to find highconfidence association rules. We demonstrate theoretically that our method is superior to exact methods when the threshold for "interesting similarity/confidence" is above the average pairwise similarity/confidence, and the average support is not too low. Our method is particularly good when transactions contain many items. We confirm in experiments on standard association mining benchmarks that this gives a significant speedup on real data sets (sometimes much larger than the theoretical guarantees). Reductions in computation time of over an order of magnitude, and significant savings in space, are observed.

Efficient Mining of Top Correlated Patterns Based on Null-Invariant Measures

“…In [19], authors derive an upper bound of Kulczynski, which was shown to be effective only for the comparatively high minimum support thresholds. The techniques based on sampling were recently proposed in [4], which are much faster, but at the cost of the incompleteness of results. Our approach works well for all null-invariant measures including Kulczynski and Cosine, which did not have efficient algorithms for low support, and it produces the complete results.…”

On Finding Similar Items in a Stream of Transactions

“…The overlap coefficient measure has the property that finding pairs having similarity over a certain threshold implies finding all association rules with confidence over that the same threshold. As argued in [18], [19], Jaccard similarity can be handled via dice similarity. parameter s determines the space usage of the algorithm, which is O(n + s) words.…”

“…For itemsets of size two (or more) the paper lacks a theoretical analysis of the proposed algorithm, but claims an empirical space usage bounded by m 3 /k 3 . Sampling according to the similarity: Our algorithms builds on top of an idea presented in [18], [19]. The sampling technique used in that algorithm is such that pairs are sampled a number of times that is proportional to their similarity.…”

“…(A more technical explanation can be found in section III-A where we improve the sampling procedure to make it suitable for a streaming environment.) The algorithms presented in [18], [19] have near-optimal running time, when no information on the distribution of similarities are given. As a matter of fact, the running time is linear in the size of the input and output (when there are many pairs of roughly the same similarity).…”

“…We base our technique on the sampling method of the BISAM algorithm [18], [19]. For each transaction the pairs are sampled according to their support, such that the pair {i, j} is sampled with probability τ f (|S i |, |S j |), where f is a function that depends on the similarity measure considered, and τ is a parameter that is used to control the sampling rate.…”