MDLdroidLite: A Release-and-Inhibit Control Approach to Resource-Efficient Deep Neural Networks on Mobile Devices

Gu, Tao; Zhang, Xi

doi:10.1109/tmc.2021.3062575

Cited by 9 publications

(10 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, many works focus on reducing the overall system resources required for DNN training [6, 11, 18, 21, 31-33, 36, 43, 51, 70, 73, 78, 87, 92, 96, 102]. For example, researchers control the layerwise growth of the model structure to enable efficient DNN training on mobile phones [104]. Other methods optimize sparse activations and redundant weights to avoid unnecessary storage of activations and weight updates during DNN training [5,28,58].…”

Section: Efficient Deep Learning Systemsmentioning

confidence: 99%

LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platforms

Kwon,

Chauhan,

Jia

et al. 2023

Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems

View full text Add to dashboard Cite

Continual Learning (CL) allows applications such as user personalization and household robots to learn on the fly and adapt to context. This is an important feature when context, actions, and users change. However, enabling CL on resource-constrained embedded systems is challenging due to the limited labeled data, memory, and computing capacity.In this paper, we propose LifeLearner, a hardware-aware meta continual learning system that drastically optimizes system resources (lower memory, latency, energy consumption) while ensuring high accuracy. Specifically, we (1) exploit meta-learning and rehearsal strategies to explicitly cope with data scarcity issues and ensure high accuracy, (2) effectively combine lossless and lossy compression to significantly reduce the resource requirements of CL and rehearsal samples, and (3) developed hardware-aware system on embedded and IoT platforms considering the hardware characteristics.As a result, LifeLearner achieves near-optimal CL performance, falling short by only 2.8% on accuracy compared to an Oracle baseline. With respect to the state-of-the-art (SOTA) Meta CL method, LifeLearner drastically reduces the memory footprint (by 178.7×), end-to-end latency by 80.8-94.2%, and energy consumption by 80.9-94.2%. In addition, we successfully deployed LifeLearner on two edge devices and a microcontroller unit, thereby enabling efficient CL on resource-constrained platforms where it would be impractical to run SOTA methods and the far-reaching deployment of adaptable CL in a ubiquitous manner. Code is available at https://github.com/theyoungkwon/LifeLearner. CCS CONCEPTS• Computer systems organization → Embedded and cyber-physical systems; • Human-centered computing → Ubiquitous and mobile computing.

show abstract

Section: Efficient Deep Learning Systemsmentioning

confidence: 99%

LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platforms

Kwon,

Chauhan,

Jia

et al. 2023

Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems

View full text Add to dashboard Cite

show abstract

“…For example, a NN pre-trained with the ImageNet dataset for pedestrian detection suffers 20% inference accuracy drop on the PASCAL dataset [36,79] 1 . On-device training with online data, hence, is essential for NNs to retain generality or be personalized [38,62,91].…”

Section: Offline Onlinementioning

confidence: 99%

“…However, since the pruned NN portions can never be selected again even if they may be useful [52], NN's representation power is weakened over time and becomes insufficient for difficult learning tasks. Training can also start from a small NN that gradually grows [48,91] (Figure 1 bottom-left), but the newly added NN layers are trained from scratch and require >30% extra training epochs [91].…”

Section: Offline Onlinementioning

confidence: 99%

ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection

Huang

Yang

Gao

2023

Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services

View full text Add to dashboard Cite

On-device training is essential for neural networks (NNs) to continuously adapt to new online data, but can be time-consuming due to the device's limited computing power. To speed up on-device training, existing schemes select trainable NN portion offline or conduct unrecoverable selection at runtime, but the evolution of trainable NN portion is constrained and cannot adapt to the current need for training. Instead, runtime adaptation of on-device training should be fully elastic, i.e., every NN substructure can be freely removed from or added to the trainable NN portion at any time in training. In this paper, we present ElasticTrainer, a new technique that enforces such elasticity to achieve the required training speedup with the minimum NN accuracy loss. Experiment results show that ElasticTrainer achieves up to 3.5× more training speedup in wall-clock time and reduces energy consumption by 2×-3× more compared to the existing schemes, without noticeable accuracy loss.

show abstract

“…Recent works optimize the resource usage of on-device training via pruning and quantization [34,45], tuning partial weights [9,47], memory profiling and optimization [15,61,64], as well as growing the NN on the fly [67]. All these works optimize training given resource constraints and do not focus on lifelong learning.…”

Section: Related Workmentioning

confidence: 99%

A Theoretical Perspective on Hyperdimensional Computing

ThomasAnthony¹,

DasguptaSanjoy²,

RosingTajana³

2021

jair

View full text Add to dashboard Cite

Hyperdimensional (HD) computing is a set of neurally inspired methods for obtaining highdimensional, low-precision, distributed representations of data. These representations can be combined with simple, neurally plausible algorithms to effect a variety of information processing tasks. HD computing has recently garnered significant interest from the computer hardware community as an energy-efficient, low-latency, and noise-robust tool for solving learning problems. In this review, we present a unified treatment of the theoretical foundations of HD computing with a focus on the suitability of representations for learning.

show abstract

MDLdroidLite: A Release-and-Inhibit Control Approach to Resource-Efficient Deep Neural Networks on Mobile Devices

Cited by 9 publications

References 31 publications

LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platforms

LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platforms

ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection

A Theoretical Perspective on Hyperdimensional Computing

Contact Info

Product

Resources

About