SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Models

Zhang, Linfeng; Tan, Zhanhong; Song, Jiebo; Chen, Jingwei; Bao, Chenglong; Ma, Kaisheng

doi:10.48550/arxiv.1906.03951

Cited by 7 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, they performed binarization on the k × k convolution kernels to cut down parameters. Meanwhile, [150] introduced scalable neural networks, which achieve neural network compression and acceleration simultaneously. Moreover, Li et al [151] designed an intensely-inverted residual block unit, which introduces inverted residual structure and multi-scale low-redundancy convolution kernels.…”

Section: B Shortcut Connectionsmentioning

confidence: 99%

Compacting Deep Neural Networks for Internet of Things: Methods and Applications

Zhang

Ying

Dai

et al. 2021

IEEE Internet Things J.

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) have shown great success in completing complex tasks. However, DNNs inevitably bring high computational cost and storage consumption due to the complexity of hierarchical structures, thereby hindering their wide deployment in Internet-of-Things (IoT) devices, which have limited computational capability and storage capacity. Therefore, it is a necessity to investigate the technologies to compact DNNs. Despite tremendous advances in compacting DNNs, few surveys summarize compacting-DNNs technologies, especially for IoT applications. Hence, this paper presents a comprehensive study on compacting-DNNs technologies. We categorize compacting-DNNs technologies into three major types: 1) network model compression, 2) Knowledge Distillation (KD), 3) modification of network structures. We also elaborate on the diversity of these approaches and make side-by-side comparisons. Moreover, we discuss the applications of compacted DNNs in various IoT applications and outline future directions.

show abstract

Section: B Shortcut Connectionsmentioning

confidence: 99%

Compacting Deep Neural Networks for Internet of Things: Methods and Applications

Zhang

Ying

Dai

et al. 2021

IEEE Internet Things J.

View full text Add to dashboard Cite

show abstract

“…The researchers who developed ResNet [13] first took the multigrid methods as evidence to support what is known as a residual representation for the interpretation of ResNet. Further, [20,8,44] adopted multi-resolution ideas to improve the performance of their networks. Additionally, a CNN model with a structure similar to that of the V-cycle multigrid is proposed to address volumetric medical image segmentation and biomedical image segmentation in [31,29].…”

Section: Related Workmentioning

confidence: 99%

An Interpretive Constrained Linear Model for ResNet and MgNet

He¹,

Xu²,

Zhu³

2021

Preprint

View full text Add to dashboard Cite

We propose a constrained linear data-feature-mapping model as an interpretable mathematical model for image classification using a convolutional neural network (CNN). From this viewpoint, we establish detailed connections between the traditional iterative schemes for linear systems and the architectures of the basic blocks of ResNet-and MgNet-type models. Using these connections, we present some modified ResNet models that compared with the original models have fewer parameters and yet can produce more accurate results, thereby demonstrating the validity of this constrained learning datafeature-mapping assumption. Based on this assumption, we further propose a general data-feature iterative scheme to show the rationality of MgNet. We also provide a systematic numerical study on MgNet to show its success and advantages in image classification problems and demonstrate its advantages in comparison with established networks.

show abstract

“…For VGG-16, [5] achieves a 1.98x Ops reduction for a 2% decrease in accuracy on ImageNet and show that they outperform both [10] and [21] on the same metric. [37] splits the network into multiple sections and learns classifiers that allow for early exit through the network depending on the input image processed. They achieve on average 2.17x reduction in Ops across networks on CIFAR-100 for no accuracy loss, and 1.99x reduction in Ops on ImageNet also for no accuracy loss.…”

Section: Background and Related Workmentioning

confidence: 99%

“…Frameworks -The various pruning techniques discussed above each have a unique set of hyperparameters that relate to filter ranking metrics as well as the manner in which the models are re-trained. For instance, [31] sequentially prunes and retrains on a per layer basis, while works such as [37] have to add many auxiliary layers on top of the chosen architecture in order to create and train their early exit classifiers. Distiller [41] and Mayo [39] are two state-of-the-art open-source frameworks that allow for experimentation with such pruning techniques.…”

Section: Background and Related Workmentioning

confidence: 99%

Now that I can see, I can improve: Enabling data-driven finetuning of CNNs on the edge

Rajagopal¹,

Bouganis²

2020

Preprint

View full text Add to dashboard Cite

In today's world, a vast amount of data is being generated by edge devices that can be used as valuable training data to improve the performance of machine learning algorithms in terms of the achieved accuracy or to reduce the compute requirements of the model. However, due to user data privacy concerns as well as storage and communication bandwidth limitations, this data cannot be moved from the device to the data centre for further improvement of the model and subsequent deployment. As such there is a need for increased edge intelligence, where the deployed models can be fine-tuned on the edge, leading to improved accuracy and/or reducing the model's workload as well as its memory and power footprint. In the case of Convolutional Neural Networks (CNNs), both the weights of the network as well as its topology can be tuned to adapt to the data that it processes. This paper provides a first step towards enabling CNN finetuning on an edge device based on structured pruning. It explores the performance gains and costs of doing so and presents an extensible open-source framework that allows the deployment of such approaches on a wide range of network architectures and devices. The results show that on average, data-aware pruning with retraining can provide 10.2pp increased accuracy over a wide range of subsets, networks and pruning levels with a maximum improvement of 42.0pp over pruning and retraining in a manner agnostic to the data being processed by the network.

show abstract

SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Models

Cited by 7 publications

References 0 publications

Compacting Deep Neural Networks for Internet of Things: Methods and Applications

Compacting Deep Neural Networks for Internet of Things: Methods and Applications

An Interpretive Constrained Linear Model for ResNet and MgNet

Now that I can see, I can improve: Enabling data-driven finetuning of CNNs on the edge

Contact Info

Product

Resources

About