mpi4py: Status Update After 12 Years of Development

Dalcín, Lisandro; Fang, Yao-Lung L.

doi:10.1109/mcse.2021.3083216

Cited by 200 publications

(91 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use Python with mpi4py package [33] to implement our developed HCMM scheme over Amazon EC2 clusters. To emulate the straggler effects in large-scale systems [34], we inject artificial delays 6 .…”

Section: B Experiments Using Amazon Ec2 Machinesmentioning

confidence: 99%

Coded Computation Over Heterogeneous Clusters

Reisizadeh

Prakash

Pedarsani

et al. 2019

IEEE Trans. Inform. Theory

186

View full text Add to dashboard Cite

In large-scale distributed computing clusters, such as Amazon EC2, there are several types of "system noise" that can result in major degradation of performance: system failures, bottlenecks due to limited communication bandwidth, latency due to straggler nodes, etc. On the other hand, these systems enjoy abundance of redundancy -a vast number of computing nodes and large storage capacity. There have been recent results that demonstrate the impact of coding for efficient utilization of computation and storage redundancy to alleviate the effect of stragglers and communication bottlenecks in homogeneous clusters. In this paper, we focus on general heterogeneous distributed computing clusters consisting of a variety of computing machines with different capabilities. We propose a coding framework for speeding up distributed computing in heterogeneous clusters by trading redundancy for reducing the latency of computation. In particular, we propose Heterogeneous Coded Matrix Multiplication (HCMM) algorithm for performing distributed matrix multiplication over heterogeneous clusters that is provably asymptotically optimal for a broad class of processing time distributions. Moreover, we show that HCMM is unboundedly faster than any uncoded scheme that partitions the total work load among the workers. To demonstrate how the proposed HCMM scheme can be applied in practice, we provide results from numerical studies and Amazon EC2 experiments comparing HCMM with three benchmark load allocation schemes -Uniform Uncoded, Load-balanced Uncoded, and Uniform Coded. In particular, in our numerical studies, HCMM achieves speedups of up to 73%, 56% and 42% respectively over the three benchmark schemes mentioned above. Furthermore, we carry out experiments over Amazon EC2 clusters and demonstrate how HCMM can be combined with rateless codes with nearly linear decoding complexity. In particular, we show that HCMM combined with the Luby transform (LT) codes can significantly reduce the overall execution time. HCMM is found to be up to 61%, 46% and 36% faster than the aforementioned three benchmark schemes, respectively. Additionally, we provide a generalization to the problem of optimal load allocation in heterogeneous settings, where we take into account the monetary costs associated with distributed computing clusters. We argue that HCMM is asymptotically optimal for budget-constrained scenarios as well. In particular, we characterize the minimum possible expected cost associated with a computation task over a given cluster of machines. Furthermore, we develop a heuristic algorithm for (HCMM) load allocation for the distributed implementation of budget-limited computation tasks.

show abstract

Section: B Experiments Using Amazon Ec2 Machinesmentioning

confidence: 99%

Coded Computation Over Heterogeneous Clusters

Reisizadeh

Prakash

Pedarsani

et al. 2019

IEEE Trans. Inform. Theory

186

View full text Add to dashboard Cite

show abstract

“…The source code is written in Python, with the usage of libraries such as NumPy for vector computation [18], Tensorflow for representing the GNN [19], NetworkX for representing the network state [20], the OpenAI Gym framework for representing the DRL environment [21], and MPI4PY for establishing the communication between workers [22]. The source code, together with all the training and evaluation results presented in this paper, are publicly available 1 .…”

Section: Resultsmentioning

confidence: 99%

Accelerating Deep Reinforcement Learning for Digital Twin Network Optimization with Evolutionary Strategies

Güemes-Palau¹,

Almasan²,

Xiao³

et al. 2022

Preprint

View full text Add to dashboard Cite

The recent growth of emergent network applications (e.g., satellite networks, vehicular networks) is increasing the complexity of managing modern communication networks. As a result, the community proposed the Digital Twin Networks (DTN) as a key enabler of efficient network management. Network operators can leverage the DTN to perform different optimization tasks (e.g., Traffic Engineering, Network Planning).Deep Reinforcement Learning (DRL) showed a high performance when applied to solve network optimization problems. In the context of DTN, DRL can be leveraged to solve optimization problems without directly impacting the real-world network behavior. However, DRL scales poorly with the problem size and complexity. In this paper, we explore the use of Evolutionary Strategies (ES) to train DRL agents for solving a routing optimization problem. The experimental results show that ES achieved a training time speed-up of 128 and 6 for the NSFNET and GEANT2 topologies respectively.

show abstract

“…Software: astrochem (Maret & Bergin 2015), matplotlib (Hunter 2007), mpi4py (Dalcin & Fang 2021), numpy (Harris et al 2020)…”

Section: Acknowledgementsmentioning

confidence: 99%

Chemical Feedbacks of Pebble Growth: Impacts on CO depletion and C/O ratios

Van Clepper,

Bergner,

Bosman

et al. 2022

Preprint

View full text Add to dashboard Cite

Observations of protoplanetary disks have revealed them to be complex and dynamic, with vertical and radial transport of gas and dust occurring simultaneously with chemistry and planet formation. Previous models of protoplanetary disks focused primarily on chemical evolution of gas and dust in a static disk, or dynamical evolution of solids in a chemically passive disk. In this paper, we present a new 1D method for modelling pebble growth and chemistry simultaneously. Gas and small dust particles are allowed to diffuse vertically, connecting chemistry at all elevations of the disk. Pebbles are assumed to form from the dust present around the midplane, inheriting the composition of ices at this location. We present the results of this model after 1 Myr of disk evolution around a 1M star at various locations both inside and outside of the CO snowline. We find that for a turbulent disk (α = 10 −3 ), CO is depleted from the surface layers of the disk by roughly 1-2 orders of magnitude, consistent with observations of protoplanetary disks. This is achieved by a combination of ice sequestration and decreasing UV opacity, both driven by pebble growth. Further, we find the selective removal of ice species via pebble growth and sequestration can increase gas phase C/O ratios to values of approximately unity. However, our model is unable to produce C/O values of ∼1.5-2.0 inferred from protoplanetary disk observations, implying selective sequestration of ice is not sufficient to explain C/O ratios > 1.

show abstract

mpi4py: Status Update After 12 Years of Development

Cited by 200 publications

References 11 publications

Coded Computation Over Heterogeneous Clusters

Coded Computation Over Heterogeneous Clusters

Accelerating Deep Reinforcement Learning for Digital Twin Network Optimization with Evolutionary Strategies

Chemical Feedbacks of Pebble Growth: Impacts on CO depletion and C/O ratios

Contact Info

Product

Resources

About