PruneTrain

Lym, Sangkug; Choukse, Esha; Zangeneh, Siavash; Wen, Wei; Sanghavi, Sujay; Erez, Mattan

doi:10.1145/3295500.3356156

Cited by 51 publications

(10 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• ElasticTrainer is time efficient. Compared to the existing schemes [28,34,64,67,73], it achieves up to 3.5× more training speedup in wall-clock time and reduces the training FLOPs by 60%.…”

Section: Offline Onlinementioning

confidence: 97%

“…A better choice is to adaptively adjust the trainable NN portion at runtime. NN pruning [43,65] for on-device training removes less important NN structures on the fly [50,64] (Figure 1 top-right). However, since the pruned NN portions can never be selected again even if they may be useful [52], NN's representation power is weakened over time and becomes insufficient for difficult learning tasks.…”

Section: Offline Onlinementioning

confidence: 99%

“…Based on such importance evaluation, we aim to achieve training speedup in wall-clock time instead of FLOPs (number of floating point operations) that is widely used in the existing work [64,91]. The major reason is that using FLOPs ignores the hardware accelerations for different NN operations.…”

Section: Offline Onlinementioning

confidence: 99%

See 2 more Smart Citations

ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection

Huang

Yang

Gao

2023

Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services

View full text Add to dashboard Cite

On-device training is essential for neural networks (NNs) to continuously adapt to new online data, but can be time-consuming due to the device's limited computing power. To speed up on-device training, existing schemes select trainable NN portion offline or conduct unrecoverable selection at runtime, but the evolution of trainable NN portion is constrained and cannot adapt to the current need for training. Instead, runtime adaptation of on-device training should be fully elastic, i.e., every NN substructure can be freely removed from or added to the trainable NN portion at any time in training. In this paper, we present ElasticTrainer, a new technique that enforces such elasticity to achieve the required training speedup with the minimum NN accuracy loss. Experiment results show that ElasticTrainer achieves up to 3.5× more training speedup in wall-clock time and reduces energy consumption by 2×-3× more compared to the existing schemes, without noticeable accuracy loss.

show abstract

Section: Offline Onlinementioning

confidence: 97%

Section: Offline Onlinementioning

confidence: 99%

See 1 more Smart Citation

ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection

Huang

Yang

Gao

2023

Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services

View full text Add to dashboard Cite

show abstract

“…The IPT approach performs pruning and training step iteratively, and it is an iterative and greedy selection procedure to approximately optimize the non-convex problem for finding sparse structures in neural networks. Different from traditional greedy methods, like ThiNet (Luo, Wu, and Lin 2017) and PruneTrain (Lym et al 2019), that permanently cuts off the weights and will never be restored, IPT can reconstruct part of pruned weights to alleviate accuracy degradation. Sparse Evolutionary Training (SET) (Mocanu et al 2018) proposes a prune-regrowth procedure that allows the pruned neurons and connections to recover randomly.…”

Section: Iterative Prune-train Iptmentioning

confidence: 99%

A One-Shot Reparameterization Method for Reducing the Loss of Tile Pruning on DNNs

Ino

2022

2022 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

Adaptive network pruning approach has recently drawn significant attention due to its excellent capability to identify the importance and redundancy of layers and filters and customize a suitable pruning solution. However, it remains unsatisfactory since current adaptive pruning methods rely mostly on an additional monitor to score layer and filter importance, and thus faces high complexity and weak interpretability. To tackle these issues, we have deeply researched the weight reconstruction process in iterative prune-train process and propose a Protective Self-Adaptive Pruning (PSAP) method. First of all, PSAP can utilize its own informationweight sparsity ratio -to adaptively adjust pruning ratio of layers before each pruning step. Moreover, we propose a protective reconstruction mechanism to prevent important filters from being pruned through supervising gradients and to avoid unrecoverable information loss as well. Our PSAP is handy and explicit because it merely depends on weights and gradients of model itself, instead of requiring an additional monitor as in early works. Experiments on ImageNet and CIFAR-10 also demonstrate its superiority to current works in both accuracy and compression ratio, especially for compressing with a high ratio or pruning from scratch.

show abstract

“…The main tenet of QoA is that, in an end-to-end data processing system, one must consider various trade-offs of quality of data, processing time, cost, result accuracy, underlying computing capabilities, to name just a few, based on specific analysis context. Dealing with trade-offs in ML is one of the important research directions [32,33]. In our previous work, we also have examined QoA for common ML pipelines [34].…”

Section: Understanding the Quality Trade-offs In End-to-end Bim Objecmentioning

confidence: 99%

Understanding quality of analytics trade-offs in an end-to-end machine learning-based classification system for building information modeling

2021

View full text Add to dashboard Cite

Optimizing quality trade-offs in an end-to-end big data science process is challenging, as not only do we need to deal with different types of software components, but also the domain knowledge has to be incorporated along the process. This paper focuses on methods for tackling quality trade-offs in a common data science process for classifying Building Information Modeling (BIM) elements, an important task in the architecture, engineering, and construction industry. Due to the diversity and richness of building elements, machine learning (ML) techniques have been increasingly investigated for classification tasks. However, ML-based classification faces many issues, w.r.t. vast amount of data with heterogeneous data quality, diverse underlying computing configurations, and complex integration with industrial BIM tools, in an end-to-end BIM data analysis. In this paper, we develop an end-to-end ML classification system in which quality of analytics is considered as the first-class feature across different phases, from data collection, feature processing, training to ML model serving. We present our method for studying the quality of analytics trade-offs and carry out experiments with BIM data extracted from Solibri to demonstrate the automation of several tasks in the end-to-end ML classification. Our results have demonstrated that the quality of data, data extraction techniques, and computing configurations must be carefully designed when applying ML classifications for BIM in order to balance constraints of time, cost, and prediction accuracy. Our quality of analytics methods presents generic steps and considerations for dealing with such designs, given the time, cost, and accuracy trade-offs required in specific contexts. Thus, the methods could be applied to the design of end-to-end BIM classification systems using other ML techniques and cloud services.

show abstract

PruneTrain

Cited by 51 publications

References 40 publications

ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection

ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection

A One-Shot Reparameterization Method for Reducing the Loss of Tile Pruning on DNNs

Understanding quality of analytics trade-offs in an end-to-end machine learning-based classification system for building information modeling

Contact Info

Product

Resources

About