A switched dynamical system framework for analysis of massively parallel asynchronous numerical algorithms

Abstract

In the near future, massively parallel computing systems will be necessary to solve computation intensive applications. The key bottleneck in massively parallel implementation of numerical algorithms is the synchronization of data across processing elements (PEs) after each iteration, which results in significant idle time. Thus, there is a trend towards relaxing the synchronization and adopting an asynchronous model of computation to reduce idle time. However, it is not clear what is the effect of this relaxation on the stability and accuracy of the numerical algorithm.In this paper we present a new framework to analyze such algorithms. We treat the computation in each PE as a dynamical system and model the asynchrony as stochastic switching. The overall system is then analyzed as a switched dynamical system. However, modeling of massively parallel numerical algorithms as switched dynamical systems results in a very large number of modes, which makes current analysis tools available for such systems computationally intractable. We develop new techniques that circumvent this scalability issue. The framework is presented on a one-dimensional heat equation and the proposed analysis framework is verified by solving the partial differential equation (PDE) in a nVIDIA Tesla TM GPU machine, with asynchronous communication between cores. Recent literature has proposed relaxing these synchronization requirements across the PEs [4]. This potentially eliminates the overhead associated with extreme parallelism Kooktae Lee and Raktim Bhattacharya are with the

A Stabilizing Control Algorithm for Asynchronous Parallel Quadratic Programming via Dual Decomposition

“…One of the biggest concerns is the notorious scalability problem (also known as the curse of dimensionality problem) that causes computational intractability, due to extremely large numbers of switching modes η. In [20], Lee et al first addressed this issue, which is briefly explained as follows.…”

Convergence Analysis of Asynchronous Consensus in Discrete-time Multi-agent Systems with Fixed Topology

“…2) Parallel computing for fixed-point iteration -Parallel computing is widely used technique to speedup computation of fixed-point iterations. For example, in [22] and [23], onedimensional heat equation using a finite difference method is solved by parallel computing in which a certain group of grid points for the finite difference scheme is assigned to each CPU or GPU core. Thus, in parallel computing each value for the group of grid points is computed by different cores, followed by communication between cores for updating values at the boundary grid points of the group.…”

Asynchronous Parallel Computing Algorithm implemented in 1D Heat Equation with CUDA

“…For decades, it has been reported that computing performance in parallel computation can deteriorate due to the synchronization penalty necessarily accompanied by parallel implementation of the given numerical scheme. Thus, there is a trend to relax this synchronization latency by adopting alternative approaches and techniques such as relaxed synchronization [1,2] or asynchronous parallel computing algorithm [3,4,5,6,7,8]. Although the asynchronous parallel computing algorithm has arisen to overcome the synchronization bottleneck, and hence speed up the computation, the randomness of asynchrony incurs unpredictability of the solution, which in turn leads to numerical inaccuracy of the solution or even instability in the worst case.…”

“…In [7], we have developed mathematical proofs for stability, rate of convergence, and error probability of asynchronous 1D heat equation via dynamical system framework (especially, the switched system framework [9,10]). All the results in this note are based on our previous research works [7]. Thus, this note aims at testing asynchronous scheme in 1D heat equation with CUDA rather than developing theory and proof.…”