MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge

Abstract

Recently, a new trend of exploring sparsity for accelerating neural network training has emerged, embracing the paradigm of training on the edge. This paper proposes a novel Memory-Economic Sparse Training (MEST) framework targeting for accurate and fast execution on edge devices. The proposed MEST framework consists of enhancements by Elastic Mutation (EM) and Soft Memory Bound (&S) that ensure superior accuracy at high sparsity ratios. Different from the existing works for sparse training, this current work reveals the importance of sparsity schemes on the performance of sparse training in terms of accuracy as well as training speed on real edge devices. On top of that, the paper proposes to employ data efficiency for further acceleration of sparse training. Our results suggest that unforgettable examples can be identified in-situ even during the dynamic exploration of sparsity masks in the sparse training process, and therefore can be removed for further training speedup on edge devices. Comparing with state-of-the-art (SOTA) works on accuracy, our MEST increases Top-1 accuracy significantly on ImageNet when using the same unstructured sparsity scheme. Systematical evaluation on accuracy, training speed, and memory footprint are conducted, where the proposed MEST framework consistently outperforms representative SOTA works. Our codes are publicly available at: https://github.com/boone891214/MEST.Recently, a new trend of exploring sparsity for training acceleration of neural networks has emerged to embrace the promising training-on-the-edge paradigm. The first works in this direction use the pruning-at-initialization approach such as SNIP [9] and GraSP [10] that first obtains a fixed sparse model structure and then follows with a traditional training process. However, the whole process is still computation-and memory-intensive, and therefore not compatible with the end-to-end edge training paradigm. Such a sparse training methodology with the pre-fixed structure also faces the problem of compromised accuracy.

Automatic Mapping of the Best-Suited DNN Pruning Schemes for Real-Time Mobile Acceleration

“…Pattern-based pruning [50,59,78] alleviates the shortcomings of prior works by incorporating the benefits from fine-grained pruning while maintaining structures that can be exploited for hardware accelerations with the help of compiler. Pattern-based pruning is a combination of kernel pattern pruning and connectivity pruning as shown in Fig.…”

Federated Learning for Internet of Things: Applications, Challenges, and Opportunities

“…To address this challenge, we envision that the lightweighted machine learning engine is needed to reduce the memory consumption for on-device training. For example, FedGKT [30], MEST [37], and TinyFL [38] are potential methods to reduce the training memory footprint for efficient on-device learning. To avoid catastrophic forgetting we also envision the use of clustering approaches that identify and store few core datasets from each time interval or leveraging IoT "Hubs" that can store non-sensitive/public datasets to inject memory in the training system.…”