Enabling Cloud Interoperability with COMPSs

Marozzo, Fabrizio; Lordan, Francesc; Rafanell, Roger; Lezzi, Daniele; Talia, Domenico; Badía, Rosa M.

doi:10.1007/978-3-642-32820-6_4

Cited by 14 publications

(8 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…COMPSs is a framework in constant development, it is receiving several new extensions and APIs to fit Big Data requirements, that vary from cloud connectors [9] to a resource manager integration [10]. In a recent work [3], COMPSs Storage API is presented, an official software interface that allows COMPSs applications and COMPSs runtime to work with persistent objects.…”

Section: Related Workmentioning

confidence: 99%

Upgrading a high performance computing environment for massive data processing

Ponce

Santos

Meira

et al. 2019

J Internet Serv Appl

Self Cite

View full text Add to dashboard Cite

High-performance computing (HPC) and massive data processing (Big Data) are two trends that are beginning to converge. In that process, aspects of hardware architectures, systems support and programming paradigms are being revisited from both perspectives. This paper presents our experience on this path of convergence with the proposal of a framework that addresses some of the programming issues derived from such integration. Our contribution is the development of an integrated environment that integretes (i) COMPSs, a programming framework for the development and execution of parallel applications for distributed infrastructures; (ii) Lemonade, a data mining and analysis tool; and (iii) HDFS, the most widely used distributed file system for Big Data systems. To validate our framework, we used Lemonade to create COMPSs applications that access data through HDFS, and compared them with equivalent applications built with Spark, a popular Big Data framework. The results show that the HDFS integration benefits COMPSs by simplifying data access and by rearranging data transfer, reducing execution time. The integration with Lemonade facilitates COMPSs's use and may help its popularization in the Data Science community, by providing efficient algorithm implementations for experts from the data domain that want to develop applications with a higher level abstraction.

show abstract

Section: Related Workmentioning

confidence: 99%

Upgrading a high performance computing environment for massive data processing

Ponce

Santos

Meira

et al. 2019

J Internet Serv Appl

Self Cite

View full text Add to dashboard Cite

show abstract

“…In addition, the DMCF's runtime differs from that of ClowdFlows and EScience Central because it is able to parallelize the execution of the tasks of each workflow, an important feature to ensure scalable data analysis workflows execution on the Cloud. COMPSs [8] is a programming model and an execution runtime, whose main objective is to ease the development of workflows for distributed environments, including private and public Clouds. With COMPSs, users create a sequential application and specify which methods of the application code will be executed remotely.…”

Section: Related Workmentioning

confidence: 99%

Scalable script-based data analysis workflows on clouds

Marozzo

Talia

Trunfio

2013

Proceedings of the 8th Workshop on Workflows in Support of Large-Scale Science

Self Cite

View full text Add to dashboard Cite

Data analysis workflows are often composed by many concurrent and compute-intensive tasks that can be efficiently executed only on scalable computing infrastructures, such as HPC systems, Grids and Cloud platforms. The use of Cloud services for the scalable execution of data analysis workflows is the key feature of the Data Mining Cloud Framework (DMCF), which provides a Web interface to develop data analysis applications using a visual workflow formalism. In this paper we describe how we extended DMCF to support also the design and execution of script-based data analysis workflows on Clouds. We introduce a workflow language, named JS4Cloud, that extends JavaScript to support the implementation of Cloud-based data analysis tasks and the handling of data on the Cloud. We also describe how data analysis workflows programmed through JS4Cloud are processed by DMCF to make parallelism explicit and to enable their scalable execution on Clouds. Finally, we present a data analysis application developed with JS4Cloud, and the performance results obtained executing the application with DMCF on the Windows Azure platform.

show abstract

“…COMPSs [12] is a programming model and an execution runtime, whose main objective is to ease the development of workflows for distributed environments including private and public Clouds. With COMPSs, users create a sequential application and specify which methods of the application code will be executed remotely.…”

Section: Related Workmentioning

confidence: 99%

“…Afterwards, n instances of J48 and m instances of JRip are executed, where each instance uses a different combination of its parameters to analyze the training set (lines 5-10). Line 11 specifies the test set, which is used to evaluate the two arrays of models generated by the n J48 instances and the m JRip instances (lines [12][13][14][15]. Then, k unlabeled datasets are specified as input, with k D 4 (line 17).…”

Section: Ensemble Learning Workflowmentioning

confidence: 99%

JS4Cloud: script‐based workflow programming for scalable data analysis on cloud platforms

Marozzo

Talia

Trunfio

2015

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

SUMMARYWorkflows are an effective paradigm to model complex data analysis processes, such as knowledge discovery in databases applications, which can be efficiently executed on distributed computing systems such as a Cloud platform. Data analysis workflows can be designed through visual programming, which is a convenient design approach for high-level users. On the other hand, script-based workflows are a useful alternative to visual workflows, because they allow expert users to program complex applications more effectively. In order to provide Cloud users with an effective script-based data analysis workflow formalism, we designed the JS4Cloud language. The main benefits of JS4Cloud are as follows: (i) it extends the well-known JavaScript language while using only its basic functions (arrays, functions, and loops); (ii) it implements both a data-driven task parallelism that automatically spawns ready-to-run tasks to the Cloud resources and data parallelism through an array-based formalism; and (iii) these two types of parallelism are exploited implicitly so that workflows can be programmed in a fully sequential way, which frees users from duties like work partitioning, synchronization, and communication. We describe how JS4Cloud has been integrated within the data mining cloud framework (DMCF), a system supporting the scalable execution of data analysis workflows on Cloud platforms. In particular, we describe how data analysis workflows modeled as JS4Cloud scripts are processed by DMCF by exploiting parallelism to enable their scalable execution on Clouds. Finally, we present some data analysis workflows developed with JS4Cloud and the performance results obtained by executing such workflows on DMCF.

show abstract

Enabling Cloud Interoperability with COMPSs

Cited by 14 publications

References 7 publications

Upgrading a high performance computing environment for massive data processing

Upgrading a high performance computing environment for massive data processing

Scalable script-based data analysis workflows on clouds

JS4Cloud: script‐based workflow programming for scalable data analysis on cloud platforms

Contact Info

Product

Resources

About