The Circulate architecture: avoiding workflow bottlenecks caused by centralised orchestration

Abstract

As the number of services and the size of data involved in workflows increases, centralised orchestration techniques are reaching the limits of scalability. In the classic orchestration model, all data passes through a centralised engine, which results in unnecessary data transfer, wasted bandwidth and the engine to become a bottleneck to the execution of a workflow.This paper presents and evaluates the Circulate architecture which maintains the robustness and simplicity of centralised orchestration, but facilitates choreography by allowing services to exchange data directly with one another. Circulate could be realised within any existing workflow framework, in this paper, we focus on WS-Circulate, a Web services based implementation.Taking inspiration from the Montage workflow, a number of common workflow patterns (sequence, fan-in and fanout), input to output data size relationships and network configurations are identified and evaluated. The performance analysis concludes that a substantial reduction in communication overhead results in a 2-4 fold performance benefit across all patterns. An end-to-end pattern through the Montage workflow results in an 8 fold performance benefit and demonstrates how the advantage of using the Circulate architecture increases as the complexity of a workflow grows.

Workflow Partitioning and Deployment on the Cloud Using Orchestra

“…The service keyword is used to declare the service identifiers s1, s2, s3, s4, s5 and s6 through lines [8][9][10][11][12][13]. Similarly, the service ports p1, p2, p3, p4, p5 and p6 are declared using the port keyword through lines [14][15][16][17][18][19]. The input and output keywords define the workflow interface, which provides an input a and an output x of the same type through lines 20-23.…”

Direct data transfer between SOAP web services in orchestration

“…Triggers and proxies are suggested as databrokers between orchestrator and component services in [8] and [6], respectively. They are responsible for buffering the intermediate data and for invoking the component services on behalf of the coordinator.…”

Dynamic Handling for Cooperating Scientific Web Services

“…Circulate [2] is a web service choreographic and orchestration system which decentralizes web service choreography through a system of proxies which aids the web services to directly talk to each other without going through a central coordinator. Orchestration is still centralized and is only used to control the overall execution.…”