A Bi-layered Parallel Training Architecture for Large-Scale Convolutional Neural Networks

Abstract

Benefitting from large-scale training datasets and the complex training network, Convolutional Neural Networks (CNNs) are widely applied in various fields with high accuracy. However, the training process of CNNs is very time-consuming, where large amounts of training samples and iterative operations are required to obtain high-quality weight parameters. In this paper, we focus on the time-consuming training process of large-scale CNNs and propose a Bi-layered Parallel Training (BPT-CNN) architecture in distributed computing environments. BPT-CNN consists of two main components: (a) an outer-layer parallel training for multiple CNN subnetworks on separate data subsets, and (b) an inner-layer parallel training for each subnetwork. In the outer-layer parallelism, we address critical issues of distributed and parallel computing, including data communication, synchronization, and workload balance. A heterogeneousaware Incremental Data Partitioning and Allocation (IDPA) strategy is proposed, where large-scale training datasets are partitioned and allocated to the computing nodes in batches according to their computing power. To minimize the synchronization waiting during the global weight update process, an Asynchronous Global Weight Update (AGWU) strategy is proposed. In the inner-layer parallelism, we further accelerate the training process for each CNN subnetwork on each computer, where computation steps of convolutional layer and the local weight training are parallelized based on task-parallelism. We introduce task decomposition and scheduling strategies with the objectives of thread-level load balancing and minimum waiting time for critical paths. Extensive experimental results indicate that the proposed BPT-CNN effectively improves the training performance of CNNs while maintaining the accuracy.

Distributed Deep Learning Model for Intelligent Video Surveillance Systems with Edge Computing

“…Various distributed AI and DL algorithms were proposed in distributed computing, cloud computing, fog computing, and edge computing environments to improve their performance and scalability [15][16][17][18][19][20]. In our previous work, we proposed a two-layer parallel CNN training architecture in a distributed computing cluster [15]. Li et al discussed the application of Machine Learning (ML) in smart industry and introduced an efficient manufacture inspection system using fog computing [18].…”

“…Focusing on distributed VS systems and AI algorithms, most current VS systems rely on traditional centralized or cloud-based solutions, facing huge data communication overhead, high latency, and severe packet loss limitations [3,12]. Existing studies have proposed various distributed AI and Deep Learning (DL) algorithms in distributed computing clusters and cloud computing platforms, such as distributed CNN, DNN, and LSTM [9,15,16]. There are many exploration spaces for distributed AI algorithms and VS system in EC environments [4,9,17].…”

A Learning Approach for Physical Layer Authentication Using Adaptive Neural Network

“…While the analysis performed provided valuable insight, the proposed algorithm required a lot of memory to store records. There has been a lot of research works on deep learning, for example, a Bi-layered Parallel Training Convolutional Neural Networks (BPT-CNN) has been proposed in a distributed computing environment [36], [37]. The work in [38] proposed a deep learning scheme for intelligent video surveillance systems with edge computing.…”

Bi-Directional Recurrent Attentional Topic Model

“…Thus, in our article, we aim to incorporate the attention mechanism into the bi-directional recurrent Bayesian topic model for documents. For training models on large-scale data, various methods such as [13,14,30,39,48,60,76] are investigated. We also propose an online learning algorithm to handle large-scale document streams.…”