NETIBIS: an efficient and dynamic communication system for heterogeneous grids

Aumage, Olivier; Hofman, Rutger F. H.; Bal, Henri E.

doi:10.1109/ccgrid.2005.1558683

Cited by 10 publications

(10 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the user may want to use a different protocol stack for example for connection coming from nodes on the same cluster reachable through a highperformance network and for connections that cross an insecure and slow WAN. Dynamically assembled component stack-This is the approach used in PadicoTM [10] and NetIbis [7]. Both parties agree on the fly on the protocol stack to use.…”

Section: Controlling the Assemblymentioning

confidence: 99%

See 1 more Smart Citation

Meta-communications in component-based communication frameworks for grids

Denis

2007

Cluster Comput

View full text Add to dashboard Cite

Applications are faced with several networkrelated problems on current grids: heterogeneous networks, firewalls, NAT, private IP addresses, non-routed networks, performance problems on WAN. Moreover, the requirements concerning communications are varied and the acceptable tradeoffs highly depends on the applications. A solution to reach the flexibility regarding communication on grids is the use of a component-based communication framework. The users then compose their own protocol stacks by assembling building blocks in the way they want. However, a truly flexible and dynamic component-based communication framework needs a meta-communication channel for its out-of-band communications required by dynamic component assembly in a consistent way on multiple nodes. The meta-communication channel is useful for some "brokered" communication methods, too, and in particular those designed to cross firewalls. The meta-communication channel has often been the "weakest link" of component-based communication frameworks: bottleneck for the performance, back-door from the security point of view, and limited connectivity.In this article, we present an architecture for a metacommunication channel that suffers from none of the aforementioned limitations. It exhibits good properties regarding connectivity, security and performance. Thus, the gain in flexibility brought by software components may be fully exploited without trading anything against flexibility.

show abstract

Section: Controlling the Assemblymentioning

confidence: 99%

“…Such a flexible assembled protocol stack based on "building blocks" has been implemented in particular in x-kernel [18], Globus XIO [8], NetIbis [7] and PadicoTM [10].…”

Section: Motivation and Principlesmentioning

confidence: 99%

Meta-communications in component-based communication frameworks for grids

Denis

2007

Cluster Comput

View full text Add to dashboard Cite

show abstract

“…In addition, there are several performance optimization parameters which can be tweaked with MPWide but not with ZeroMQ. Additionally, the NetIBIS [3] and the PadicoTM [5] tools provide functionalities similar to MPWide, though NetIBIS is written in Java, which is not widely supported on the compute nodes of supercomputers, and PadicoTM requires the presence of a central rendez-vous server. For fast file transfers, alternatives include GridFTP and various closed-source file transfer software solutions.…”

Section: Software Metapapermentioning

confidence: 99%

“…The supercomputers were interconnected by a lightpath with 10 Gigabit/s bandwidth capacity. Our main simulation consisted of 2048 3 particles, and required about 10% of its runtime to exchange data over the wide area network.…”

Section: The Cosmogrid Projectmentioning

confidence: 99%

MPWide: a light-weight library for efficient message passing over wide area networks

Groen

Rieder

Zwart

2013

Journal of Open Research Software

View full text Add to dashboard Cite

Groen et al 2013 MPWide: a light-weight library for efficient message passing over wide area networks. Journal of Open Research Software, 1: e9, DOI: http://dx.doi.org/10.5334/jors.ah We present MPWide, a light weight communication library which allows efficient message passing over a distributed network. MPWide has been designed to connect application running on distributed (super) computing resources, and to maximize the communication performance on wide area networks for those without administrative privileges. It can be used to provide message-passing between application, move files, and make very fast connections in client-server environments. MPWide has already been applied to enable distributed cosmological simulations across up to four supercomputers on two continents, and to couple two different bloodflow simulations to form a multiscale simulation. Journal of open research softwareGroen et al Art. e9, p. 2 of 8 network of supercomputers. The complexity of the underlying supercomputing and network architectures, as well as the high communication performance required for the CosmoGrid project, required us to develop a library that was both highly configurable and trivial to install, regardless of the underlying (super)computing platform.There are a number of tools which have similarities to MPWide. ZeroMQ [1], is a socket library which supports a wide range of platforms. However, compared with MPWide it does have a heavier dependency footprint. Among other things it depends on uuid-dev, a package that requires administrative privileges to install. In addition, there are several performance optimization parameters which can be tweaked with MPWide but not with ZeroMQ. Additionally, the NetIBIS [3] and the PadicoTM [5] tools provide functionalities similar to MPWide, though NetIBIS is written in Java, which is not widely supported on the compute nodes of supercomputers, and PadicoTM requires the presence of a central rendez-vous server. For fast file transfers, alternatives include GridFTP and various closed-source file transfer software solutions. There are also dedicated tools for running MPI applications across clusters [11,14,2] and for coupling applications to form a multiscale simulation (e.g., MUSCLE [4] and the Jungle Computing System [6]). Summary of research using MPWideMPWide has been applied to support several research and technical projects so far. In this section we summarize these projects, the purpose for which MPWide has been used in these projects, and the performance that we obtained using MPWide. The CosmoGrid projectMPWide has been used extensively in the CosmoGrid project, for which it was originally developed. In this project we required a library that enabled fast message passing between supercomputers and which was trivial to install on PCs, clusters, little Endian Cray-XT4 supercomputers and big Endian IBM Power6 supercomputers. In addition, we needed MPWide to deliver solid communication performance over light paths and dedicated 10Gbps networks, even when these networks were not...

show abstract

“…The IBIS project [6] is also developing software oriented towards multiplatform Grid computing in Java that allows sophisticated interprocess communication.…”

Section: Related Workmentioning

confidence: 99%

PyMW: a Python Module for Parallel Master Worker Computing

Heien¹,

Kornafeld²,

Takata³

et al.

Proceedings of the First International Conference on Parallel, Distributed and Grid Computing for Engineering

View full text Add to dashboard Cite

We describe a general purpose master-worker parallel computation Python module called PyMW. PyMW provides a unified interface to multiple computation environments including multicore processors, networked clusters and the Berkeley Open Infrastructure for Network Computing (BOINC) software platform. PyMW is intended to support rapid development, testing and deployment of large scale master-worker style computations. It is also designed to allow easy extension to other computing environments with little change in the master-worker program. We demonstrate the effectiveness and scalability of PyMW by performing several master-worker style parallel computations on a multicore machine, a networked cluster and a BOINC project.

show abstract

NETIBIS: an efficient and dynamic communication system for heterogeneous grids

Cited by 10 publications

References 22 publications

Meta-communications in component-based communication frameworks for grids

Meta-communications in component-based communication frameworks for grids

MPWide: a light-weight library for efficient message passing over wide area networks

PyMW: a Python Module for Parallel Master Worker Computing

Contact Info

Product

Resources

About