MapReduce as a programming model for association rules algorithm on Hadoop

Yang, Xin; Liu, Zhen; Fu, Yan

doi:10.1109/icicis.2010.5534718

Cited by 95 publications

(33 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Especially when frequent item sets are higher or need to have a date mining update, the algorithm has higher efficiency and feasibility. In [18], an improved Apriori algorithm based on MapReduce mode is described, which can handle massive datasets with a large number of nodes on Hadoop platform. In [19], authors realize the parallelization of Apriori algorithm for the massive Web log mining, and verify the efficiency of Apriori algorithm which has been improved by parallelization.…”

Section: Introductionmentioning

confidence: 99%

A New Improved Apriori Algorithm Based on Compression Matrix

Luo

2014

Advanced Data Mining and Applications

View full text Add to dashboard Cite

Abstract. The existing Apriori algorithm based on matrix still has the problems that the candidate itemsets are too large and matrix takes up too much memory space. To solve these problems, an improved Apriori algorithm based on compression matrix is proposed. The improvement ideas of this algorithm are as follows: (1) reducing the times of scanning matrix set during compressing by adding two arrays to record the counts of 1 in the row and column; (2) minimizing the scale of matrix and improving space utilization by deleting the itemsets which cannot be connected and the infrequent itemsets in compressing matrix; (3) decreasing the errors of the mining result by changing the condition of deleting the unnecessary transaction column;(4) reducing the cycling number of algorithm by changing the stopping condition of program. Instance analysis and experimental results show that the proposed algorithm can accurately and efficiently mines all frequent itemsets in transaction database, and improves the efficiency of mining association rules.

show abstract

Section: Introductionmentioning

confidence: 99%

A New Improved Apriori Algorithm Based on Compression Matrix

Luo

2014

Advanced Data Mining and Applications

View full text Add to dashboard Cite

show abstract

“…Several studies [17,28,45,27,20,23] have been done for mining frequent patterns in distributed environments, inspired by the MapReduce framework proposed by…”

Section: Related Workmentioning

confidence: 99%

“…Some of them [17,28,45] use a naive approach which computes the support of every itemset in the dataset in a single MapReduce round, resulting in huge data replication. An adaption of FP-Growth algorithm to MapReduce, called PFP [27], is a more sophisticated approach.…”

Section: Related Workmentioning

confidence: 99%

Approximate Parallel High Utility Itemset Mining

Chen

2016

Big Data Research

View full text Add to dashboard Cite

High utility itemset mining discovers itemsets whose utility is above a given threshold, where utilities measure the importance of itemsets. In high utility itemset mining, memory and time performance limitations cause scalability issues, when the dataset is very large. In this thesis, the problem is addressed by proposing a distributed parallel algorithm, PHUI-Miner, and a sampling strategy, which can be used either separately or simultaneously. PHUI-Miner parallelizes the state-of-theart high utility itemset mining algorithm HUI-Miner. The sampling strategy investigates the required sample size of a dataset, in order to achieve a given accuracy.We also propose an approach combining sampling with PHUI-Miner, which provides better time performance. In our experiments, we show that PHUI-Miner has high performance and outperforms the state-of-the-art non-parallel algorithm. The sampling strategy achieves accuracies much higher than the guarantee. Extensive experiments are also conducted to compare the time performance of PHUI-Miner with and without sampling.ii

show abstract

“…Transactions are allocated to Mappers and frequent k-itemsets are extracted from each Mapper before the results are shuffled through combiners and the final k-itemsets are extracted according to support and confidence thresholds. In Oruganti et al [4], Kovacs et al [5], Li et al [6], Mappers and Reducers are used, but in [7,8], in addition to Mappers and Reducers, Combiners are used for better shuffling and to address performance issues.…”

Section: Related Workmentioning

confidence: 99%