A Parallel Distributed Weka Framework for Big Data Mining Using Spark

Koliopoulos, Aris-Kyriakos; Yiapanis, Paraskevas; Tekiner, Firat; Nenadić, Goran; Keane, John

doi:10.1109/bigdatacongress.2015.12

Cited by 50 publications

(18 citation statements)

References 17 publications

(17 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…They evaluated on strong and weak scaling workloads on 5GB, 20GB and 80GB datasets and 8, 32 and 128 cores systems. Furthermore, in case of large datasets, Spark approach linearity shows the strong scaling efficiencies [18].…”

Section: Intrusion Detection System On Big Data Using Deep Learning Tmentioning

confidence: 99%

Intrusion Detection System on Big data using Deep Learning Techniques

Dahiya*,

Srivastva

2020

IJITEE

View full text Add to dashboard Cite

Big data is the huge amount of data with different types of V’s: Velocity, Variety as well as Volume. It can be semi-structured, unstructured or structured, due to which it is not easy to analyze the data. To extract the hidden knowledge and to detect the attacks on large amount of data new architecture, techniques, algorithms, and analytics are required. Using traditional techniques to detect attacks is very difficult. In this paper, the detailed review has been done on intrusion detection on various fields using deep learning and gives an idea of applications of deep learning. The number of attacks has been increased in computer networks. A powerful Intrusion Detection System (IDS) is required to ensure the security of a network. Based on review, it is found that some studies have been done in this field, but a deep and exhaustive work has still not been done. Many researchers proposed an IDS using deep learning for unforeseen and unpredictable attacks but not for Big Data. The proposed work is based on Deep learning based intrusion detection System for big datasets named hybrid-DeepResNet-RNN run till 1,000 epochs with learning rate varying range [0.01-0.5] and three ensemble techniques, Random Forest, Decision tree regression and Gradient Boosting Tree (GBT). It is used to develop the hybrid, secure, scalable NIDS which is based on deep learning and big data techniques. The proposed classifiers produce a more reliable classification than a single classifier. The experimental results are in terms of detection rate (98.86%), false positive rate (1.110%), accuracy (99.34%) and F-Measure (97.90%). The results illuminate the better performance than existing anomaly detection techniques in the big data environment.

show abstract

Section: Intrusion Detection System On Big Data Using Deep Learning Tmentioning

confidence: 99%

Intrusion Detection System on Big data using Deep Learning Techniques

Dahiya*,

Srivastva

2020

IJITEE

View full text Add to dashboard Cite

show abstract

“…As informational indexes become bigger and progressively confused, an outrageous learning machine (ELM) that keeps running in a customary sequential condition can't understand its capacity to be quick and successful [2]. Viable Big Data Mining requires adaptable and proficient arrangements that are additionally effectively available to clients at all degrees of mastery [20]. Huge Data is "high-volume, high-speed, or potentially high assortment data resources that require new types of preparing to empower upgraded basic leadership, knowledge revelation and procedure enhancement" [1].…”

Section: Introductionmentioning

confidence: 99%

“…Utilizing worked in libraries independently, existing frameworks have figured out how to utilize Spark for such examination of Big Data [16]. There are as of now two LDA frameworks on Spark, to be specific, Spark LDA [19] and the official one in ML lib [20] (which uses variational induction as opposed to CGS). In any case, these frameworks perform and scale ineffectively, since them two have high computational expense [9].…”

Section: Introductionmentioning

confidence: 99%

The Deep Learning and Apache Spark Enabled Architecture for Improving the Performance of Big Data Classification

Brahmane¹,

Krishna²

2019

IJITEE

View full text Add to dashboard Cite

At present the Big Data applications, for example, informal communication, therapeutic human services, horticulture, banking, financial exchange, instruction, Facebook and so forth are producing the information with extremely rapid. Volume and Velocity of the Big information assumes a significant job in the presentation of Big information applications. Execution of the Big information application can be influenced by different parameters. Expediently search, proficiency and precision are the a portion of the overwhelming parameters which influence the general execution of any Big information applications. Due the immediate and aberrant inclusion of the qualities of 7Vs of Big information, each Big Data administrations anticipate the elite. Elite is the greatest test in the present evolving situation. In this paper we propose the Big Data characterization way to deal with speedup the Big Data applications. This paper is the review paper, we allude different Big information advancements and the related work in the field of Big Data Classification. In the wake of learning and understanding the writing we discover the holes in existing work and techniques. Finally we propose the novel methodology of Big Data characterization. Our methodology relies on the Deep Learning and Apache Spark engineering. In the proposed work two stages are appeared; first stage is include choice and second stage is Big Data Classification. Apache Spark is the most reasonable and predominant innovation to execute this proposed work. Apache Spark is having two hubs; introductory hubs and last hubs. The element choice will be occur in introductory hubs and Big Data Classification will happen in definite hubs of Apache Spark.

show abstract

“…Such a distributed and parallelized processing led to high scalability, high efficiency, fault-tolerance and load balancing of huge size data sets. Google Firstly introduced MapReduce as a programming model and Apache Hadoop from Yahoo is considered the first implementation of MapReduce that becomes the most popular platform of large scale distributed data processing [1]. Granular computing has emerged as a new rapidly growing information processing paradigm under the umbrella of Computational Intelligence.…”

Section: Introductionmentioning

confidence: 99%

A Novel Data Mining Approach for Big Data Based on Rough Sets and Fuzzy Logic

Abdelrahman¹,

Hefny²

2019

IJIES

View full text Add to dashboard Cite

The term "Big Data" is a buzzword which describes new technologies that manipulate very large data sets which are massively generated by heterogonous sources. This new term encourages data scientists to extend their work and modify their techniques to overcome the new challenges come with big data concepts. Granular computing has emerged as a new rapidly growing information processing paradigm inside the community of Computational Intelligence. Theories of Fuzzy sets and Rough sets theory are considered powerful examples of granular computing that can be applied to data mining techniques to extract nontrivial knowledge from huge data. The aim of this paper is to introduce a data mining approach for big data based on integrating fuzzy sets and rough sets theories. The proposed approach provides a novel granular data mining for big data that allow extracting useful knowledge and rules from huge data to enhance the decision making process. The proposed approach has been applied on different types of datasets. The experimental results show that our proposed approach is more efficient and robust when dealing with very big datasets and obtained consistent classification rules with classification accuracy 100%.

show abstract

A Parallel Distributed Weka Framework for Big Data Mining Using Spark

Cited by 50 publications

References 17 publications

Intrusion Detection System on Big data using Deep Learning Techniques

Intrusion Detection System on Big data using Deep Learning Techniques

The Deep Learning and Apache Spark Enabled Architecture for Improving the Performance of Big Data Classification

A Novel Data Mining Approach for Big Data Based on Rough Sets and Fuzzy Logic

Contact Info

Product

Resources

About