Modeling and simulation of distributed computing workflows in heterogeneous network environments

Abstract

Next-generation computation- and network-intensive collaborative applications in various science, engineering, and e-commerce fields feature large-scale computing workflows of complex structures. Efficient algorithms are needed for task scheduling, module deployment, and service provisioning to support the execution of such distributed workflows in heterogeneous network environments and optimize their end-to-end performance for fast system response or smooth data flow. However, deploying large-scale distributed applications in real network environments is extremely challenging due to the inherent dynamics in the reliability, availability, accessibility, and capacity of massively distributed system resources, which are typically shared among a broad community of users over the Internet or dedicated connections. We propose a simulation system to study the execution dynamics of distributed computing workflows and evaluate the network performance of workflow scheduling or mapping algorithms before actual deployment and experimentation. The proposed simulation system visually illustrates the dynamic execution process of workflows in network environments by simulating module execution on computer nodes and data transfer over network links in a completely distributed and parallel manner. Furthermore, the simulation system takes background traffic and workload into consideration to achieve a high level of simulation accuracy for distributed applications deployed in shared production network environments. We implement the simulation system using multi-threaded programming and conduct extensive testings on various mapping schemes using a large number of simulated workflows and networks. The simulation-based performance measurements are quantitatively confirmed by both the experimental observations collected in real networks and the theoretical results obtained by rigorous performance analysis based on well-defined mathematical models.

QMAEA: A quantum multi-agent evolutionary algorithm for multi-objective combinatorial optimization

“…14 Along with the development of information, many projects on MS have been carried out, and a large number of research papers in different areas have been published. Although different areas have different ways of simulation, there are a few common research contents emphasized the most: architecture of MS, 15–19 verification, validation and accreditation (VVA) of MS, 20,21 communication of distributed MS, 22,23 resource management and workflow optimization in MS 24–28 and so on.…”

“…Among these hot research points, simulation optimization such as simulation resource and service allocation and composition is one of the fundamental factors which directly decided the efficiency of MS systems. 29–33 With the flourishing of large-scaled distributed simulation and the introduction of service-oriented technologies, various distributed simulation resources need to be connected as services and corresponding tasks are implemented in collaboration. Other than in the uniformed distributed computing platform, the properties, quality of service (QoS) and mutual relations among simulation services are more complicated.…”