Efficient Mining of Top Correlated Patterns Based on Null-Invariant Measures

Abstract

Abstract. Mining strong correlations from transactional databases often leads to more meaningful results than mining association rules. In such mining, null (transaction)-invariance is an important property of the correlation measures. Unfortunately, some useful null-invariant measures such as Kulczynski and Cosine, which can discover correlations even for the very unbalanced cases, lack the (anti)-monotonicity property. Thus, they could only be applied to frequent itemsets as the post-evaluation step. For large datasets and for low supports, this approach is computationally prohibitive. This paper presents new properties for all known null-invariant measures. Based on these properties, we develop efficient pruning techniques and design the Apriori-like algorithm NICOMINER for mining strongly correlated patterns directly. We develop both the threshold-bounded and the top-k variations of the algorithm, where top-k is used when the optimal correlation threshold is not known in advance and to give user control over the output size. We test NICOMINER on real-life datasets from different application domains, using Cosine as an example of the null-invariant correlation measure. We show that NICOMINER outperforms support-based approach more than an order of magnitude, and that it is very useful for discovering top correlations in itemsets with low support.

A data mining correlated patterns-based periodic decentralized replication strategy for data grids

“…In addition, they do not distinguish the different requests issued from the different grid jobs and consider them as successive file accesses. Various studies have shown the limits of association rule mining based on the support and confidence approach [8,9,38,39]. They indeed confirm that it results on an excessively large number of rules, with a large part of them either redundant or not reflecting the true correlation relationship among data [13,43].…”

Inferring Knowledge from Concise Representations of Both Frequent and Rare Jaccard Itemsets

“…Experiments realised on several datasets show the efficiency of GMJP according to both quantitative and qualitative aspects. An important direction for future work is to extend our approach to other correlation measures [10,18,20,22] through classifying them into classes of measures sharing the same properties.…”

“…In [26], the authors provide a unified definition of existing null-invariant correlation measures and propose the GAMINER approach allowing the extraction of frequent high correlated patterns according to the Cosine and to the Kulczynski measures. In this same context, the NICOMINER algorithm was also proposed in [10] and it allows the extraction of correlated patterns according to the Cosine measure. In this same context, we cite also the AETHERIS approach [21] which allow the extraction of condensed representation of correlated patterns according to user's preferences.…”

Efficient mining of correlated sequential patterns based on null hypothesis

“…We admit that the failure of Apriori property on correlation bring challenges in computational issues, but on the other hand it makes the mining results more effective and meaningful. Therefore, we re-examine various interestingness measures [2,29,14], in search of a measure we could use to mine correlated sequential patterns, and finally obtain the motivation of designing our correlation measure from the null hypothesis in Ngram testing.…”

“…In Table 4, we list top-ranked sequential patterns (whose size is larger than two) according to four measures: support (see Definition 2.2), all-confidence [29], lift [14], and cor (Equation 1). We can see that patterns with the highest support values are mostly random combinations of popular words: even though some phrases make sense as high level concepts, e.g., 'database system', their useless duplicates may appear multiple times, such as, 'oriented database system', 'data base system', and 'object database system'.…”