Workflows for science: a challenge when facing the convergence of HPC and Big Data

Abstract

Workflows have been traditionally a mean to describe and implement the computing experiments, usually parametric studies and explorations searching for the best solution, that scientific researchers want to perform. A workflow is not only the computing application, but a way of documenting a process. Science workflows may be of very different nature depending on the area of research, matching the actual experiment that the scientist want to perform. Workflow Management Systems are environments that offer the researchers tools to define, publish, execute and document their workflows.In some cases, the science workflows are used to generate data; in other cases are used to analyse existing data; only in a few cases, workflows are used both to generate and analyse data. The design of experiments is in some cases generated blindly, without a clear idea of which points are relevant to be computed/simulated, ending up with huge amount of computation that is performed following a brute-force strategy.However, the evolution of systems and the large amount of data generated by the applications require an in-situ analysis of the data, thus requiring new solutions to develop workflows that includes both the simulation/computational part and the analytic part. What is more, the fact that both components, computation and analytics, can be run together will enable the possibility of defining more dynamic workflows, with new computations being decided by the analytics in a more efficient way.The first part of the paper will review current approaches that a set of scientific communities follow in the development of their workflows. This paper does not intent to be exhaustive in the compilation of different approaches available to develop and deploy workflows. We focus on the Workflow Management Systems used by a set of scientific communities and their representative use cases, with the objective of understanding their different needs and requirements. The second part of the paper proposes a new software architecture to develop a new family of end-to-end workflows that enables the management of dynamic workflows composed of simulations, analytics and visualization, including inputs/outputs from streams.Keywords: workflows, scientific applications, Big Data. IntroductionWorkflows appeared last century and have been used in the manufacturing industry as a mean to optimize their processes. Examples of traditional (non-IT) workflows can be found in the assembly lines, i.e., the Ford Model T assembly line standardized the production processes and was the first continuous delivery pipeline for the automotive industry. This process reduced the costs of manufacturing from $850 to $260 in 1924.The time and motion studies defined by Taylor [52] and Gilbreth [41] had significant impact in the manufacturing processes. These studies proposed to break manufacturing activities into small, simple steps, to determine with accuracy the amount of time required to perform each of the steps. Then, the sequence of movements taken by the employee has...

End-to-End Workflows for Climate Science: Integrating HPC Simulations, Big Data Processing, and Machine Learning

“…Most systems typically support workflows defined as Directed Acyclic Graphs (DAGs) of tasks, where no cyclic dependencies among tasks are allowed, although some solutions also support cyclic interaction among them [23]. WMSs also differ in terms of interfaces provided for workflow development and submission, including graphical user interfaces (GUIs), textual interfaces and more programmatic ones (APIs) [3]. Moreover, workflow systems provide capabilities such as resilience and fault detection, optimized scheduling/execution of tasks, provenance tracking, workflow validation and monitoring, which are some of the key functionalities for supporting large-scale workflows [14].…”

“…Current scientific workflows, however, do not typically integrate simulation-centric and data-centric aspects of research due to their very different, sometimes orthogonal, infrastructure requirements [2]. End-to-end workflow solutions, capable of handling the whole workflow from numerical model simulation to data processing and visualization, would represent very valuable solutions for speeding up the research process and improving scientists' productivity [3]. Such solutions would also allow supporting execution on software stacks with different computing paradigms (i.e., High Performance Computing (HPC) and Cloud).…”

Unveiling User Behavior on Summit Login Nodes as a User

“…Scientific workflows are used almost universally across scientific domains for solving complex and large-scale computing and data analysis problems. The importance of workflows is highlighted by the fact that they have underpinned some of the most significant discoveries of the past few decades [3]. Many of these workflows have significant demands for computation, storage, and communcation, and thus they have been increasingly executed on large-scale computer systems [14].…”

Micro-Workflows Data Stream Processing Model for Industrial Internet of Things

“…However, still, each traditional workflow working in batch execution mode over tightly coupled tasks. Another challenge is that the features of data visualization and data stream input/output are limited support in current WFMSs [5]. Such complexity increased in case of fog computing systems, where the execution environment itself decoupled over multiple separated geographical locations.…”

“…But a number of challenges appear with WFMSs, for example, the tasks of the workflow are tightly coupled by means of intricate dependencies between them [35]. Also, the features of data visualization and data stream input/output are limited support in current WFMSs [5]. Thus in [2,27], the concept of Micro-Workflows has been presented.…”