A lightweight communication library for distributed computing

Groen, Derek; Rieder, Steven; Grosso, Paola; Laat, Cees de; Zwart, Simon Portegies

doi:10.1088/1749-4699/3/1/015002

Cited by 14 publications

(14 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, later runs resulted in more stable performance once we used a different path and adjusted the MPWide configuration to use very small TCP buffer sizes per stream [4]. Reproduced from Groen et al [27]. Note: The communication time was relatively high (∼ 35% of total runtime)) in this test run due to the small problem size, in [27] we also present results from a test run with 2048 3 particles, which had a communication overhead of ∼ 13% of total runtime.…”

Section: B Establishing a Distributed Supercomputing Infrastructurementioning

confidence: 99%

From Thread to Transcontinental Computer: Disturbing Lessons in Distributed Supercomputing

Groen

Zwart

2015

2015 IEEE 11th International Conference on E-Science

Self Cite

View full text Add to dashboard Cite

Abstract-We describe the political and technical complications encountered during the astronomical CosmoGrid project. CosmoGrid is a numerical study on the formation of large scale structure in the universe. The simulations are challenging due to the enormous dynamic range in spatial and temporal coordinates, as well as the enormous computer resources required. In CosmoGrid we dealt with the computational requirements by connecting up to four supercomputers via an optical network and make them operate as a single machine. This was challenging, if only for the fact that the supercomputers of our choice are separated by half the planet, as three of them are located scattered across Europe and fourth one is in Tokyo. The co-scheduling of multiple computers and the 'gridification' of the code enabled us to achieve an efficiency of up to 93% for this distributed intercontinental supercomputer. In this work, we find that high-performance computing on a grid can be done much more effectively if the sites involved are willing to be flexible about their user policies, and that having facilities to provide such flexibility could be key to strengthening the position of the HPC community in an increasingly Cloud-dominated computing landscape. Given that smaller computer clusters owned by research groups or university departments usually have flexible user policies, we argue that it could be easier to instead realize distributed supercomputing by combining tens, hundreds or even thousands of these resources.

show abstract

Section: B Establishing a Distributed Supercomputing Infrastructurementioning

confidence: 99%

From Thread to Transcontinental Computer: Disturbing Lessons in Distributed Supercomputing

Groen

Zwart

2015

2015 IEEE 11th International Conference on E-Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…To optimize speed over wide area networks, it has a local buffer of 3 MB and it will prefer sending over receiving up to the point that it will not allow more incoming data if the send buffers are too large or numerous. The MPWide 1.8 [21] library is optionally enabled for connections between MTO's. MPWide is a library to optimize message-passing performance over wide-area networks, especially for larger messages.…”

Section: A1 Implementation Of the Muscle Transport Overlay (Mto)mentioning

confidence: 99%

Distributed multiscale computing with MUSCLE 2, the Multiscale Coupling Library and Environment

Borgdorff

Mamonski

Bosak

et al. 2014

Journal of Computational Science

View full text Add to dashboard Cite

We present the Multiscale Coupling Library and Environment: MUSCLE 2. This multiscale component-based execution environment has a simple to use Java, C++, C, Python and Fortran API, compatible with MPI, OpenMP and threading codes. We demonstrate its local and distributed computing capabilities and compare its performance to MUSCLE 1, file copy, MPI, MPWide, and GridFTP. The local throughput of MPI is about two times higher, so very tightly coupled code should use MPI as a single submodel of MUSCLE 2; the distributed performance of GridFTP is lower, especially for small messages. We test the performance of a canal system model with MUSCLE 2, where it introduces an overhead as small as 5% compared to MP

show abstract

“…In MAPPER, we have defined a tool chain [ 33 ] to compute multiscale models that can be described with the MMSF. It starts by specifying the architecture with the Multiscale Modelling Language (MML) [ 13 ] in a dedicated user interface and then executing it with the GridSpace Experiment Workbench [ 32 ] for acyclic coupling topologies, and MUSCLE 2 [ 30 ], if needed in combination with MPWide [ 31 ], for cyclic coupling topologies. Distributed multiscale simulations are coordinated by middleware, in our case QCG-Broker [ 34 ] and the Application Hosting Environment [ 35 ].…”

Section: Multiscale Modelling and Simulation Frameworkmentioning

confidence: 99%

“…In MAPPER, we have chosen MUSCLE 2 [ 30 ] and MPWide [ 31 ] as coupling technologies for cyclic models, where submodels must communicate frequently, and the GridSpace Experiment Workbench (EW) [ 32 , 33 ] for acyclic coupling topologies. These technologies have local and distributed computing capabilities.…”

Section: Introductionmentioning

confidence: 99%

Performance of distributed multiscale simulations

Borgdorff

Belgacem

Bona-Casas

et al. 2014

Phil. Trans. R. Soc. A.

View full text Add to dashboard Cite

Multiscale simulations model phenomena across natural scales using monolithic or component-based code, running on local or distributed resources. In this work, we investigate the performance of distributed multiscale computing of component-based models, guided by six multiscale applications with different characteristics and from several disciplines. Three modes of distributed multiscale computing are identified: supplementing local dependencies with large-scale resources, load distribution over multiple resources, and load balancing of small- and large-scale resources. We find that the first mode has the apparent benefit of increasing simulation speed, and the second mode can increase simulation speed if local resources are limited. Depending on resource reservation and model coupling topology, the third mode may result in a reduction of resource consumption.

show abstract

A lightweight communication library for distributed computing

Cited by 14 publications

References 22 publications

From Thread to Transcontinental Computer: Disturbing Lessons in Distributed Supercomputing

From Thread to Transcontinental Computer: Disturbing Lessons in Distributed Supercomputing

Distributed multiscale computing with MUSCLE 2, the Multiscale Coupling Library and Environment

Performance of distributed multiscale simulations

Contact Info

Product

Resources

About