Asynchronous Parallel Computing

Abstract

Many single‐core statistical algorithms do not scale well to learn the large‐scale data that we encounter. With the emergence of multiple core central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), and supercomputers, parallel algorithms are developed to take advantage of multiple cores. However, parallel computing brings new challenges and bottlenecks. One major bottleneck is the synchronization among all the cores. Synchronization requires the cores that finish early to wait for other cores, and the waiting time could be very long when the number of cores is large. Asynchronous parallel computing lets the cores continue without waiting for others. However, the naive way of merely removing the synchronization step does not work in many cases. Adapting one algorithm to the asynchronous parallel environment is not trivial and usually requires careful reformulation. Moreover, the convergence of these asynchronous algorithms is challenging to show. In this article, we introduce several ways to implement asynchronous statistical algorithms. More specifically, we introduce asynchronous parallel coordinate algorithms that update the coordinates in parallel and asynchronous stochastic algorithms that evaluate the data samples in parallel.