Distributed Learning of CNNs on Heterogeneous CPU/GPU Architectures

Abstract

Convolutional Neural Networks (CNNs) have shown to be powerful classification tools in tasks that range from check reading to medical diagnosis, reaching close to human perception, and in some cases surpassing it. However, the problems to solve are becoming larger and more complex, which translates to larger CNNs, leading to longer training times-the computational complex part-that not even the adoption of Graphics Processing Units (GPUs) could keep up to. This problem is partially solved by using more processing units and distributed training methods that are offered by several frameworks dedicated to neural network training, such as Caffe, Torch or TensorFlow. However, these techniques do not take full advantage of the possible parallelization offered by CNNs and the cooperative use of heterogeneous devices with different processing capabilities, clock speeds, memory size, among others. This paper presents a new method for the parallel training of CNNs that can be considered as a particular instantiation of model parallelism, where only the convolutional layer is distributed. In fact, the convolutions processed during training (forward and backward propagation included) represent from 60-90% of global processing time. The paper analyzes the influence of network size, bandwidth, batch size, number of devices, including their processing capabilities, and other parameters. Results show that this technique is capable of diminishing the training time without affecting the classification performance for both CPUs and GPUs. For the CIFAR-10 dataset, using 1

Violence Detection in Videos by Combining 3D Convolutional Neural Networks and Support Vector Machines

“…In fact, in our implementation, a large portion of the computation time was due to the memory load of the C3D network, instead of the real output computation. Therefore, we can assert that, by improving the management of the memory in the proposed implementation, this approach can also be used for real-time violence detection, especially when coupled with dedicated software or codesigned hardware (Chen et al 2018;Marques, Falcao, and Alexandre 2018).…”