Realizing Neural Decoder at the Edge With Ensembled BNN

Vikas, Devannagari; Nayak, Nancy; Kalyani, Sheetal

doi:10.1109/lcomm.2021.3102319

Cited by 3 publications

(1 citation statement)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Before that let us first discuss the number of parameters and FLOPs in a single convolution layer. The memory and FLOPs for 1D-CNN and 2D-CNN were given in [27] and [21] respectively. Considering L layers in a ResNet architecture, the trainable parameters are given by φ = {W 1 , .…”

Section: Saving In Memory and Computation Using Rrelumentioning

confidence: 99%

Rotate the ReLU to implicitly sparsify deep networks

Nayak¹,

Kalyani²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

In the era of Deep Neural Network based solutions for a variety of real-life tasks, having a compact and energy-efficient deployable model has become fairly important. Most of the existing deep architectures use Rectifier Linear Unit (ReLU) activation. In this paper, we propose a novel idea of rotating the ReLU activation to give one more degree of freedom to the architecture. We show that this activation wherein the rotation is learned via training results in the elimination of those parameters/filters in the network which are not important for the task. In other words, rotated ReLU seems to be doing implicit sparsification. The slopes of the rotated ReLU activations act as coarse feature extractors and unnecessary features can be eliminated before retraining. Our studies indicate that features always choose to pass through a lesser number of filters in architectures such as ResNet and its variants. Hence, by rotating the ReLU, the weights or the filters that are not necessary are automatically identified and can be dropped thus giving rise to significant savings in memory and computation. Furthermore, in some cases, we also notice that along with saving in memory and computation we also obtain improvement over the reported performance of the corresponding baseline work in the popular datasets such as MNIST, CIFAR-10, CIFAR-100, and SVHN. I. INTRODUCTIONMachine Learning has gained a lot of attention recently for surpassing human-level performance in solving problems starting from real-life applications to complex tasks. It leverages the

show abstract

Section: Saving In Memory and Computation Using Rrelumentioning

confidence: 99%