Coding for Large-Scale Distributed Machine Learning

Abstract

This article aims to give a comprehensive and rigorous review of the principles and recent development of coding for large-scale distributed machine learning (DML). With increasing data volumes and the pervasive deployment of sensors and computing machines, machine learning has become more distributed. Moreover, the involved computing nodes and data volumes for learning tasks have also increased significantly. For large-scale distributed learning systems, significant challenges have appeared in terms of delay, errors, efficiency, etc. To address the problems, various error-control or performance-boosting schemes have been proposed recently for different aspects, such as the duplication of computing nodes. More recently, error-control coding has been investigated for DML to improve reliability and efficiency. The benefits of coding for DML include high-efficiency, low complexity, etc. Despite the benefits and recent progress, however, there is still a lack of comprehensive survey on this topic, especially for large-scale learning. This paper seeks to introduce the theories and algorithms of coding for DML. For primal-based DML schemes, we first discuss the gradient coding with the optimal code distance. Then, we introduce random coding for gradient-based DML. For primal–dual-based DML, i.e., ADMM (alternating direction method of multipliers), we propose a separate coding method for two steps of distributed optimization. Then coding schemes for different steps are discussed. Finally, a few potential directions for future works are also given.

New Classification Method for Independent Data Sources Using Pawlak Conflict Model and Decision Trees

“…The distributed models process all or part of the data at different nodes [ 3 , 4 ]. A solution in which all the data are simultaneously aggregated and stored in a single set is both inefficient and often impossible to apply [ 5 ]. Therefore, most research papers have proposed a collaborative solution without data aggregation.…”

Information Theoretic Methods for Future Communication Systems

“…The topics touched upon include a multi-layer grant-free transmission method [ 1 ], a direct transform-coding approach that maps the delay-Doppler domain to the time domain [ 2 ], degree-of-freedom bounds for multi-antenna, multi-user, and frequency-selective interference channels with an instantaneous relay with or without coordination [ 3 ], new coded caching methods to reduce latency with user cooperation and simultaneous transmission [ 4 ], and a low-resolution downlink precoding method for multi-input single-output channels with orthogonal frequency-division multiplexing [ 5 ]. Furthermore, machine learning methods are discussed in the context of knowledge graphs for semantic communications [ 6 ] and in a review of the state-of-the-art coding methods for large-scale distributed machine learning [ 7 ]. Focusing on coding theory over rings, a new weight that extends the traditional Hamming weight used for algebraic structures is proposed and its properties are analyzed in [ 8 ].…”