Mining Frequent Itemsets with Vertical Data Layout in MapReduce

Jen, Tao-Yuan; Marinica, Claudia; Ghariani, Abir

doi:10.1007/978-3-319-38901-1_5

Cited by 5 publications

(6 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Jen, T. Y., et al have created a novel vertical format based parallel method for finding frequent patterns called Apriori_V with MapReduce platform. They proved that it provides a significant improvement in reducing the number of operations and decreasing computational complexity [14]. The authors in [15] have introduced a Parallel Regular Frequent Pattern (PRF) method to find out the regular-frequent patterns from large databases using VDF format and proved from the experiments that the algorithm reduced the number of database scans, I/O cost and inter-process communication.…”

Section: Related Workmentioning

confidence: 99%

A Multithread, Novel Pattern Based Algorithm for Finding Frequent Patterns With Jagged Array and Vertical Data Format

Sumathi¹,

Murugan²,

Umadevi³

2021

INDJCSE

View full text Add to dashboard Cite

Frequent pattern mining is essential for discovering hidden items from a database with more than a prescribed threshold. Knowing frequent patterns helps us to determine the relationship between the items. Many researchers narrated novel algorithms for sequential frequent itemset mining using a single thread, but still, there is a need for time, memory efficient and scalable one. Therefore, the research study proposed an approach for finding frequent patterns, namely TB-NPF-VDF (Thread Based, Novel Pattern Formations with Vertical Data Format), which uses a new way of generating candidate items to minimize the time. Also, it employs a multithread concept and runs several threads simultaneously, one for each frequent 1-itemset to generate the remaining frequent itemsets for that item. Further, it also employs a jagged array to store the frequent patterns to reduce the memory requirement. The research work has been implemented and tested using four real-time datasets. Further, it has been compared with Matrix-Apriori, VDF and NPF-VDF (without multithread), and the experimental results reveal that TB-NPF-VDF outperforms significantly in terms of runtime and storage.

show abstract

Section: Related Workmentioning

confidence: 99%

A Multithread, Novel Pattern Based Algorithm for Finding Frequent Patterns With Jagged Array and Vertical Data Format

Sumathi¹,

Murugan²,

Umadevi³

2021

INDJCSE

View full text Add to dashboard Cite

show abstract

“…The very first application of ARM was the market basket analysis. The market basket analysis identifies customers' buying patterns based on the frequent items purchased by them [Jen et al (2016)]. This descriptive method of data mining is now used in several areas other than business.…”

Section: Related Workmentioning

confidence: 99%

“…For each frequent itemset l, all nonempty subsets of l are generated. For every nonempty subset s of l, the rule"s  (l-s)" is generated if σ(l) / σ( s) ≥ min_conf [Han et al (2006)], [Jen et al (2016)].…”

Section: Rdd-eclat Algorithmsmentioning

confidence: 99%

Association Rule Mining Algorithms for Big Data using RDD-ECLAT Algorithms

Martha¹,

Vankdothu

Abdul

et al. 2021

Preprint

View full text Add to dashboard Cite

The revolution in technology for storing and processing big data leads to data intensive computing as a new paradigm. To find the valuable and precise big data knowledge, efficient and scalable data mining techniques are required. In data mining, different techniques are applied depending on the kind of knowledge to be mined. Association rules are generated from the frequent itemsets computed by frequent itemset mining (FIM) algorithms. The problem of designing scalable and efficient frequent itemset mining algorithms on the Spark RDD framework. The research done in this thesis aims to improve the performance (in terms of execution time) of the existing Spark-based frequent itemset mining algorithms and efficiently re-design other frequent itemset mining algorithms on Spark. The particular problem of interest is re-designing the Eclat algorithm in the distributed computing environment of the Spark. The paper proposes and implements a parallel Eclat algorithm using the Spark RDD architecture, dubbed RDD-Eclat. EclatV1 is the earliest version, followed by EclatV2, EclatV3, EclatV4, and EclatV5. Each version is the consequence of a different technique and heuristic being applied to the preceding variant. Following EclatV1, the filtered transaction technique is used, followed by heuristics for equivalence class partitioning in EclatV4 and EclatV5. EclatV2 and EclatV3 are slightly different algorithmically, as are EclatV4 and EclatV5. Experiments on synthetic and real-world datasets.

show abstract

“…Several frequent itemset mining algorithms have been proposed on MapReduce framework since the foundation of Hadoop. Among these proposed algorithms, majors are Apriori based [6,7] [ [12][13][14][15][16] while some are FP-Growth based [17] and very few are based on Eclat [11]. Most of the Apriori based algorithms are simply straight forward implementation of Apriori on MapReduce framework [12].…”

Section: Related Workmentioning

confidence: 99%

Performance optimization of MapReduce-based Apriori algorithm on Hadoop cluster

Singh

Garg

Mishra

2018

Computers & Electrical Engineering

View full text Add to dashboard Cite

Many techniques have been proposed to implement the Apriori algorithm on MapReduce framework but only a few have focused on performance improvement. FPC (Fixed Passes Combined-counting) and DPC (Dynamic Passes Combined-counting) algorithms combine multiple passes of Apriori in a single MapReduce phase to reduce the execution time. In this paper, we propose improved MapReduce based Apriori algorithms VFPC (Variable Size based Fixed Passes Combined-counting) and ETDPC (Elapsed Time based Dynamic Passes Combined-counting)over FPC and DPC. Further, we optimize the multi-pass phases of these algorithms by skipping pruning step in some passes, and propose Optimized-VFPC and Optimized-ETDPC algorithms. Quantitative analysis reveals that counting cost of additional un-pruned candidates produced due to skipped-pruning is less significant than reduction in computation cost due to the same. Experimental results show that VFPC and ETDPC are more robust and flexible than FPC and DPC whereas their optimized versions are more efficient in terms of execution time.

show abstract

Mining Frequent Itemsets with Vertical Data Layout in MapReduce

Abstract: International audienc

Cited by 5 publications

References 16 publications

A Multithread, Novel Pattern Based Algorithm for Finding Frequent Patterns With Jagged Array and Vertical Data Format

A Multithread, Novel Pattern Based Algorithm for Finding Frequent Patterns With Jagged Array and Vertical Data Format

Association Rule Mining Algorithms for Big Data using RDD-ECLAT Algorithms

Performance optimization of MapReduce-based Apriori algorithm on Hadoop cluster

Contact Info

Product

Resources

About