Hiddenite: 4K-PE Hidden Network Inference 4D-Tensor Engine Exploiting On-Chip Model Construction Achieving 34.8-to-16.0TOPS/W for CIFAR-100 and ImageNet

Hirose, Kazutoshi; Yu, Jongmin; Ando, Kota; Okoshi, Yasuyuki; Garcia-Arias, Angel Lopez; Suzuki, Junnosuke; Chu, Thiem Van; Kawamura, Kazushi; Motomura, Masato

doi:10.1109/isscc42614.2022.9731668

Cited by 16 publications

(16 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To further verify the generalization ability of the D-Net algorithm model, we used the fashion_mnist dataset (Khanday et al, 2021) and the cifar_100 dataset (Hirose et al, 2022) for further experiments.…”

Section: Public Datasetsmentioning

confidence: 99%

Disaster cassification net: A disaster classification algorithm on remote sensing imagery

Yuan

Ding

Liu

et al. 2023

Front. Environ. Sci.

View full text Add to dashboard Cite

As we all know, natural disasters have a great impact on people’s lives and properties, and it is very necessary to deal with disaster categories in a timely and effective manner. In light of this, we propose using tandem stitching to create a new Disaster Cassification network D-Net (Disaster Cassification Net) using the D-Conv, D-Linear, D-model, and D-Layer modules. During the experiment, we compared the proposed method with “CNN” and “Transformer”, we found that disaster cassification net compared to CNN algorithm Params decreased by 26–608 times, FLOPs decreased by up to 21 times, Precision increased by 1.6%–43.5%; we found that disaster cassification net compared to Transformer algorithm Params decreased by 23–149 times, FLOPs decreased by 1.7–10 times, Precision increased by 3.9%–25.9%. Precision increased by 3.9%–25.9%. And found that disaster cassification net achieves the effect of SOTA(State-Of-The-Art) on the disaster dataset; After that, we compared the above-mentioned MobileNet_v2 with the best performance on the classification dataset and CCT network are compared with disaster cassification net on fashion_mnist and CIFAR_100 public datasets, respectively, and the results show that disaster cassification net can still achieve the state-of-the-art classification effect. Therefore, our proposed algorithm can be applied not only to disaster tasks, but also to other classification tasks.

show abstract

Section: Public Datasetsmentioning

confidence: 99%

Disaster cassification net: A disaster classification algorithm on remote sensing imagery

Yuan

Ding

Liu

et al. 2023

Front. Environ. Sci.

View full text Add to dashboard Cite

show abstract

“…The SLTH does not only merge training and pruning, but also quantization: strong lottery tickets have been found to be robust to binarization [34], [41], sparser when using ternary masks [42], and more accurate with a scalar mask [43]. Furthermore, sparse random weights and binary masks can be exploited for designing energy-efficient inference hardware [4], which can even switch the binary mask for adjusting computational cost at the edge [4] or for reusing the same random weights for a different task [44].…”

Section: ) the Strong Lottery Ticket Hypothesismentioning

confidence: 99%

“…All weights and biases are counted as occupying 32 bit. However, it is not necessary to store weights in the case of supermask training, since they can be generated on the fly from the original seed with a random number generator [1], [4]. Furthermore, this seed can be substituted with a hash of other model parameters [4], so it is not necessary to store it either.…”

Section: E Model Compression Schemementioning

confidence: 99%

“…However, it is not necessary to store weights in the case of supermask training, since they can be generated on the fly from the original seed with a random number generator [1], [4]. Furthermore, this seed can be substituted with a hash of other model parameters [4], so it is not necessary to store it either. Therefore, the model size of models trained with supermask training is only the size of the supermaskone bit per weight-in addition to the memory size of the affine normalization parameters, if any.…”

Section: E Model Compression Schemementioning

confidence: 99%

“…This claim supersedes the Lottery Ticket Hypothesis [3], as weight training is entirely unnecessary. These sparse, random, and tiny subnetworks achieve competitive performance in vision tasks and can be exploited for very efficient inference hardware implementation [4].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Recurrent Residual Networks Contain Stronger Lottery Tickets

et al. 2023

Self Cite

View full text Add to dashboard Cite

Accurate neural networks can be found just by pruning a randomly initialized overparameterized model, leaving out the need for any weight optimization. The resulting subnetworks are small, sparse, and ternary, making excellent candidates for efficient hardware implementation. However, finding optimal connectivity patterns is an open challenge. Based on the evidence that residual networks may be approximating unrolled shallow recurrent neural networks, we conjecture that they contain better candidate subnetworks at inference time when explicitly transformed into recurrent architectures. This hypothesis is put to the test on image classification tasks, where we find subnetworks within the recurrent models that are more accurate and parameter-efficient than both the ones found within feedforward models and than the full models with learned weights. Furthermore, random recurrent subnetworks are tiny: under a simple compression scheme, ResNet-50 is compressed without a drastic loss in performance to 48.55× less memory size, fitting in under 2 megabytes. Code available at: https://github.com/Lopez-Angel/hidden-fold-networks.

show abstract

Bridging the Gap Between Reservoirs and Neural Networks

Hashimoto,

García-Arias,

2023

Photonic Neural Networks With Spatiotemporal Dynamics

View full text Add to dashboard Cite

While reservoir computing is drawing attention, its applications are limited to small tasks. This chapter proposes a solution to this issue by introducing the Hidden-Fold Network, a recursive model with fixed random weights that resembles reservoir computing. The model is constructed based on recent discoveries in neural networks, namely the Strong Lottery Ticket Hypothesis and folding. By pruning an overparameterized model that is randomly initialized, it is possible to find accurate neural networks without the need for weight optimization. It is conjectured that residual networks may contain better subnetwork candidates for inference time when transformed into recurrent architectures, as they may be approximating unrolled shallow recurrent neural networks. This hypothesis is tested in image classification tasks, where subnetworks within the recurrent models are found to be more accurate and parameter-efficient than those within feedforward models, as well as the full models with learned weights.

show abstract

Hiddenite: 4K-PE Hidden Network Inference 4D-Tensor Engine Exploiting On-Chip Model Construction Achieving 34.8-to-16.0TOPS/W for CIFAR-100 and ImageNet

Cited by 16 publications

References 5 publications

Disaster cassification net: A disaster classification algorithm on remote sensing imagery

Disaster cassification net: A disaster classification algorithm on remote sensing imagery

Recurrent Residual Networks Contain Stronger Lottery Tickets

Bridging the Gap Between Reservoirs and Neural Networks

Contact Info

Product

Resources

About