Implementing Apache Spark jobs execution and Apache Spark cluster creation for Openstack Sahara

Abstract

In this paper the problem of creating virtual clusters in clouds for big data analysis with Apache Hadoop and Apache Spark is discussed. Both clouds and MapReduce models are popular nowadays for a bunch of reasons: cheapness and efficient big data analysis respectively. For these thoughts, having an open source solution for building clusters is important. The article gives an overview on existing methods for Apache Spark cluster creation in clouds. We consider two open source cloud engines OpenStack and Eucalyptus and the most popular proprietary cloud service Amazon Web Services and examine cloud related features presented by these systems. Afterwards, we regard possible ways of creating virtual clusters for big data processing in OpenStack and describe their pros and cons. In the second part we describe in details one of these solutions that uses service Sahara. Sahara represents a cluster management system for OpenStack and it is used for setting up virtual clusters and executing MapReduce jobs. Sahara did not support contemporary versions of Apache Spark. The article introduces the results of our work that led to a Sahara modification, describes its idea and implementation details. By virtue of our modification, Sahara is able to create and use virtual clusters with contemporary versions of Apache Spark in OpenStack clouds.

Optimizing Parallelism of Big Data Analytics at Distributed Computing System

“…We show a specific example 'Word count' which is one of the most common data analytics in table II. The system proceeds the analytics' tasks related to map procedure with 'map' and 'flatMap' API supported by the framework and reduce procedure with 'reduceByKey' and 'collect' API [16]. # Save and load model clusters.save(sc,"target/org/apache/spark/PythonKMeans Example/KMeansModel") sameModel = KMeansModel.load(sc, "target/org/apache/spark/ PythonKMeansExample/KMeansModel") Table III shows an example 'K-means' which is more complex than 'Word count'.…”