FaaS Orchestration of Parallel Workloads

Barcelona-Pons, Daniel; Garcı́a-López, Pedro; Ruiz, Álvaro; Gómez-Gómez, Amanda; Parı́s, Gerard; Sánchez-Artigas, Marc

doi:10.1145/3366623.3368137

Cited by 25 publications

(31 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Serverless computing application and design. Researchers have predicted the future of serverless computing [39,62] and have evaluated its efficacy across several domains, including scientific computing [87], real-time systems [78], publisher-subscriber systems [55], internet-of-things [82], edge computing [56], and big data processing [42,72,93]. While this thread of research confirms the potential of serverless infrastructure, another thread focusses on improving the state-of-the-art serverless design.…”

Section: Related Workmentioning

confidence: 94%

Workflow Integration Alleviates Identity and Access Management in Serverless Computing

Sankaran

Datta

Bates

2020

Annual Computer Security Applications Conference

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 94%

Workflow Integration Alleviates Identity and Access Management in Serverless Computing

Sankaran

Datta

Bates

2020

Annual Computer Security Applications Conference

View full text Add to dashboard Cite

“…Two previous papers [4,5] have compared public FaaS or-chestration services for coordinating massively parallel workloads. In those studies, IBM Composer offered the fastest performance and reduced overheads to execute map jobs whereas ASF or ADF imposed considerable overheads.…”

Section: Related Workmentioning

confidence: 99%

“…In that case, it would call the available clients with the updated model and the cycle would start again. If it decides that the training has finished, it can notify the controller server by, for example, sending a request to a specific endpoint, containing the final model (5).…”

Section: Specialized Workflows: Federated Learning Orchestratormentioning

confidence: 99%

“…The reality is that existing serverless orchestration systems are not designed for long-running data analytics tasks [4,5]. Either they are focused on short-running highly interactive workflows (Amazon Express Workflows, IBM Composer) or im-Email addresses: aitor.arjona@urv.cat (Aitor Arjona), pedro.garcia@urv.cat (Pedro García López), josep.sampe@urv.cat (Josep Sampé), aslom@us.ibm.com (Aleksander Slominski), villard@us.ibm.com (Lionel Villard) pose considerable overheads for synchronizing massively parallel jobs (Azure Durable Functions, Amazon Step Functions, Google Cloud Workflows).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Triggerflow: Trigger-based orchestration of serverless workflows

Arjona

Garcı́a-López

Sampé

et al. 2021

Future Generation Computer Systems

View full text Add to dashboard Cite

As more applications are being moved to the Cloud thanks to serverless computing, it is increasingly necessary to support the native life cycle execution of those applications in the data center.But existing cloud orchestration systems either focus on short-running workflows (like IBM Composer or Amazon Step Functions Express Workflows) or impose considerable overheads for synchronizing massively parallel jobs (Azure Durable Functions, Amazon Step Functions). None of them are open systems enabling extensible interception and optimization of custom workflows.We present Triggerflow: an extensible Trigger-based Orchestration architecture for serverless workflows. We demonstrate that Triggerflow is a novel serverless building block capable of constructing different reactive orchestrators (State Machines, Directed Acyclic Graphs, Workflow as code, Federated Learning orchestrator). We also validate that it can support high-volume event processing workloads, auto-scale on demand with scale down to zero when not used, and transparently guarantee fault tolerance and efficient resource usage when orchestrating long running scientific workflows.

show abstract

“…Currently, Fork-Join work ow cannot be composed using any of the available serverless orchestration services [4] like Amazon Step Functions (ASF) (December 2016), Azure Durable Functions (June 2017), and IBM Composer (October 2017). These services lack the ability to dynamically launch functions in parallel.…”

Section: Introductionmentioning

confidence: 99%

ReactiveFnJ: A Choreographed Model for Fork-Join Workflow in Serverless Computing

Bharti¹,

Goel

Gupta

2022

Preprint

View full text Add to dashboard Cite

Function-as-a-Service (FaaS) is an event-based reactive programming model where functions run in ephemeral stateless containers for short duration. For building complex serverless applications, function composition is crucial to coordinate and synchronize the workflow of an application. Some serverless orchestration systems exist, but they are in their primitive state and do not provide inherent support for non-trivial workflows like, Fork-Join. To address this gap, we propose a fully serverless and scalable design model ReactiveFnJ for Fork-Join workflow. The intent of this work is to illustrate a design which is completely choreographed, reactive, asynchronous, and represents a dynamic composition model for serverless applications. Our design uses two innovative patterns, namely, Relay Composition and Master-Worker Composition to solve execution time-out challenges. As a Proof-of-Concept (PoC), the prototypical implementation of Split-Sort-Merge use case, based on Fork-Join workflow is discussed and evaluated. The ReactiveFnJ handles embarrassingly parallel computations, and its design does not depend on any external orchestration services, messaging services, and queue services. ReactiveFnJ facilitates in designing fully automated pipelines for distributed data processing systems, satisfying the Serverless Trilemma in true essence. A file of any size can be processed using our effective and extensible design without facing execution time-out challenges. The proposed model is generic and can be applied to a wide range of serverless applications that are based on the Fork-Join workflow pattern. It fosters the choreographed serverless composition for complex workflows. The proposed design model is useful for software engineers and developers in industry and commercial organizations, total solution vendors and academic researchers.

show abstract

FaaS Orchestration of Parallel Workloads

Cited by 25 publications

References 4 publications

Workflow Integration Alleviates Identity and Access Management in Serverless Computing

Workflow Integration Alleviates Identity and Access Management in Serverless Computing

Triggerflow: Trigger-based orchestration of serverless workflows

ReactiveFnJ: A Choreographed Model for Fork-Join Workflow in Serverless Computing

Contact Info

Product

Resources

About