Research on Parallel Support Vector Machine Based on Spark Big Data Platform

Abstract

With the development of cloud computing and distributed cluster technology, the concept of big data has been expanded and extended in terms of capacity and value, and machine learning technology has also received unprecedented attention in recent years. Traditional machine learning algorithms cannot solve the problem of effective parallelization, so a parallelization support vector machine based on Spark big data platform is proposed. Firstly, the big data platform is designed with Lambda architecture, which is divided into three layers: Batch Layer, Serving Layer, and Speed Layer. Secondly, in order to improve the training efficiency of support vector machines on large-scale data, when merging two support vector machines, the “special points” other than support vectors are considered, that is, the points where the nonsupport vectors in one subset violate the training results of the other subset, and a cross-validation merging algorithm is proposed. Then, a parallelized support vector machine based on cross-validation is proposed, and the parallelization process of the support vector machine is realized on the Spark platform. Finally, experiments on different datasets verify the effectiveness and stability of the proposed method. Experimental results show that the proposed parallelized support vector machine has outstanding performance in speed-up ratio, training time, and prediction accuracy.

Scheduling of Jobs Allocation for Apache Spark Using Kubernetes for Efficient Execution of Big Data Application

“…Its popularity has grown since initial support for running spark on kubernetes was added to apache spark [22,31]. The reasons are due to better isolation and resource distribution for simultaneous Spark applications in Kubernetes and the benefits of using a homogeneous and cloud-native setup for the entire technology stack of the enterprise.…”

Performance Optimization of Machine Learning Algorithms Based on Spark

“…Literature [18] proposed a Dynamic Quantum Particle Swarm Optimization (DQPSO) algorithm to seek the optimal parameters to improve the performance of MBSVM, and through a large number of simulation experiments, the proposed DQPSO algorithm is proved to be better than the traditional QPSO algorithm. Literature [19] conceived a parallelized support vector machine based on the Spark big data platform to change the status quo that traditional machine algorithms are helpless in parallelization problems. Experiments based on different datasets confirm that the proposed parallelized support vector machine performs better in terms of acceleration ratio, training time, and prediction accuracy.…”

Surrogate-based optimization algorithm in application of design of ship dimensions

“…In order to solve the problem of complex optimization objectives, this paper will use the Surrogate model [12][13][14] to solve this problem. The specific implementation of the Surrogate model is realized by the support vector machine [15,16] provided by the LibSVM toolbox [17][18][19] .…”