A Distributed Method for Fast Mining Frequent Patterns From Big Data

Huang, Pengyu; Cheng, Wan-Shu; Chen, Ju-Chin; Chung, Wen-Yu; Chen, Young-Lin; Lin, Kun‐Wei

doi:10.1109/access.2021.3115514

Cited by 8 publications

(4 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DFP [35] is a parallel version of Eclat that was introduced as a distributed approach to increase the mining efficiency for association rules. The transmission cost will be increased if the communication between clients (nodes) increased.…”

Section: Vertical Layout-based Algorithmsmentioning

confidence: 99%

An Efficient Spark-Based Hybrid Frequent Itemset Mining Algorithm for Big Data

Al-Bana

Salah

Othman

2022

Data

View full text Add to dashboard Cite

Frequent itemset mining (FIM) is a common approach for discovering hidden frequent patterns from transactional databases used in prediction, association rules, classification, etc. Apriori is an FIM elementary algorithm with iterative nature used to find the frequent itemsets. Apriori is used to scan the dataset multiple times to generate big frequent itemsets with different cardinalities. Apriori performance descends when data gets bigger due to the multiple dataset scan to extract the frequent itemsets. Eclat is a scalable version of the Apriori algorithm that utilizes a vertical layout. The vertical layout has many advantages; it helps to solve the problem of multiple datasets scanning and has information that helps to find each itemset support. In a vertical layout, itemset support can be achieved by intersecting transaction ids (tidset/tids) and pruning irrelevant itemsets. However, when tids become too big for memory, it affects algorithms efficiency. In this paper, we introduce SHFIM (spark-based hybrid frequent itemset mining), which is a three-phase algorithm that utilizes both horizontal and vertical layout diffset instead of tidset to keep track of the differences between transaction ids rather than the intersections. Moreover, some improvements are developed to decrease the number of candidate itemsets. SHFIM is implemented and tested over the Spark framework, which utilizes the RDD (resilient distributed datasets) concept and in-memory processing that tackles MapReduce framework problem. We compared the SHFIM performance with Spark-based Eclat and dEclat algorithms for the four benchmark datasets. Experimental results proved that SHFIM outperforms Eclat and dEclat Spark-based algorithms in both dense and sparse datasets in terms of execution time.

show abstract

Section: Vertical Layout-based Algorithmsmentioning

confidence: 99%

An Efficient Spark-Based Hybrid Frequent Itemset Mining Algorithm for Big Data

Al-Bana

Salah

Othman

2022

Data

View full text Add to dashboard Cite

show abstract

“…Second, it combines the local rules from each task using a majority-voting mechanism. [20] proposed a distributed frequent pattern mining (DFP) method that aims to improve the execution time of FP-Growth by reducing the data transmission cost between the nodes in the parallel and distributed processing frameworks, high memory cost, and redundant execution time owing to unadaptable nodes. DFP provides a set of algorithms for providing data and workloads in a fast and scalable manner, as well as a data structure for storing items with their counts to reduce network data transmission.…”

Section: Related Workmentioning

confidence: 99%

Towards Enhancing the Performance of Parallel FP-Growth on Spark

2022

View full text Add to dashboard Cite

Frequent itemset mining (FIM) is a crucial tool for identifying hidden patterns in information. FP-Growth is an FIM algorithm used to find associations. When the data size increases, the execution of FIM algorithms on a single machine suffers from computational problems, such as memory and time consumption. For these reasons, parallel and distributed processing on platforms such as Spark is essential. The parallel frequent pattern (PFP) is the implementation of FP-Growth in Spark. The main problem with PFP is that it does not consider the load balancing between cluster units. This research proposes an enhanced balanced parallel frequent pattern ''EBPFP'' algorithm to enhance and balance the PFP. The proposed algorithm (EBPFP) proposes two ideas. First, a strategy for load balancing between groups is proposed to ensure that the items are evenly divided between the nodes, and the cluster resources are used more effectively. Second, the improved conditional pattern base (ICPB) method aims to remove infrequent items from the conditional pattern base before constructing local FP-Trees. The experimental results show that the proposed EBPFP algorithm outperforms PFP, and the difference in running time between EBPFP and PFP was 21.56% and 39.72%, respectively.INDEX TERMS Big data, data mining, association rule analysis, frequent pattern growth algorithm, spark, load balancing.

show abstract

“…Huang et al [16] introduced a new algorithm for fast mining frequent patterns using a distributed computing system. Frequent pattern mining identifies the important patterns that are presented in a given dataset and reduce the latency rate in the analysis process.…”

Section: Related Workmentioning

confidence: 99%

“…Similarly, the services are held for 12-20 mins for a user. From this detailing, the metrics of data availability, mining time, access time, failure rate, and sharing ratio are compared with the existing UIMS [21], FCNN-DM [24], and DMFM [16] methods.…”

Section: Performance Assessmentmentioning

confidence: 99%

Data Management and Service Mode of Library Based on Data Mining Algorithm

Zhang

Chi

2022

Scientific Programming

View full text Add to dashboard Cite

Data management for large-scale data library services with mining procedures improves the availability and readiness of heterogeneous sources. The heterogeneous data sources are assimilated as a single entity through mining procedures to meet the data demands. This article introduces connectivity-persistent data mining method (CDMM) to improve the data handling precision with boosting availability. The proposed method relies on federated learning for identifying the service demands, thereby providing data mining. The learning paradigm accumulates information on shared data library existence over various services. Based on the availability, further mining demands are forwarded to the data management system. If the existence verified by the federated learning is adaptable, then sharing-enabled mining is endorsed for the connected users. The data management then augments several heterogeneous shared libraries to meet the mining requirements. This process is reversible based on the service mode and existence. Therefore, the proposed method improves data availability with less mining and access time and fewer failures.

show abstract

A Distributed Method for Fast Mining Frequent Patterns From Big Data

Cited by 8 publications

References 23 publications

An Efficient Spark-Based Hybrid Frequent Itemset Mining Algorithm for Big Data

An Efficient Spark-Based Hybrid Frequent Itemset Mining Algorithm for Big Data

Towards Enhancing the Performance of Parallel FP-Growth on Spark

Data Management and Service Mode of Library Based on Data Mining Algorithm

Contact Info

Product

Resources

About