Task-based programming in COMPSs to converge from HPC to big data

Abstract

Abstract-Task-based programming has proven to be a suitable model for high-performance computing (HPC) applications. Different implementations have been good demonstrators of this fact, and have promoted the acceptance of task-based programming in the OpenMP standard.Furthermore, in recent years, Apache Spark has gained wide popularity in business and research environments as a programming model for addressing emerging Big-Data problems. COMP Superscalar (COMPSs) is a task-based environment that tackles distributed computing (including Clouds), and is a good alternative for a task-based programming model for Big Data applications.This paper describes why we consider that task-based programming models are a good approach for Big Data applications. The paper includes a comparison of Spark and COMPSs in terms of architecture, programming model and performance. It focuses on the differences that both frameworks have in structural terms, on their programmability interface, and in terms of their efficiency by means of three widely known benchmarking kernels: Wordcount, Kmeans and Terasort. These kernels enable the evaluation of the more important functionalities of both programming models and analyse different workflows and conditions.The main results achieved from this comparison are: (1) COMPSs is able to extract the inherent parallelism from the user code with minimal coding effort as opposed to Spark, which requires the existing algorithms to be adapted and rewritten by explicitly using their pre-defined functions; (2) it is an improvement in terms of performance when compared with Spark, and (3) COMPSs has shown to scale better than Spark in most cases.Finally, we discuss the advantages and disadvantages of both frameworks, highlighting the differences that make them unique, thereby helping to choose the right framework for each particular objective.

Upgrading a high performance computing environment for massive data processing

“…A recent work [34] compared COMPSs performance in Java applications to Apache Spark, using a cluster architecture normally associated with HPC applications (e.g., low-latency networks and shared network disks). In this work, our integration allow us to take COMPSs into a cluster usually adopted in Data Science scenarios, with only traditional networking hardware and with disks distributed among the cluster nodes.…”

“…As previously mentioned in Section 2, a recent work [34] compared COMPSs performance in Java applications to Apache Spark, using a cluster architecture normally associated with HPC applications. Since our integration allows COMPSs to better interface with a cluster architecture more frequently found in Data Science scenarios, in this study we present a comparison of the two systems under those conditions, Table 4 presents a performance comparison between COMPSs and Spark using the applications Grep, Wordcount, KMeans and KNN.…”

Transparent Orchestration of Task-based Parallel Applications in Containers Platforms

“…COMPSs is used in production at MareNostrum supercomputer and has been used to implement different real world applications, specially in the area of BioInformatics and Computational Genomics [10] [13] [17], Big Data analytics [29] and as building block for several scientific cyber-infrastructures [45] [22]. Other examples of applications developed with COMPSs can be found in [7].…”

Executing linear algebra kernels in heterogeneous distributed infrastructures with PyCOMPSs

“…PySpark is a binding to the widely extended framework Spark [17]. A previous paper compares several Big Data algorithms using the native version of both COMPSs 1 and Spark runtimes [18], showing that COMPSs is able to get better or competitive results in comparison to Spark.…”