LogNet: Energy-efficient neural networks using logarithmic computation

Lee, Edward H.; Miyashita, Daisuke; Chai, Elaina; Murmann, Boris; Wong, S.S.

doi:10.1109/icassp.2017.7953288

Cited by 123 publications

(71 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, using 4 bits in linear quantization results in a 27.8% loss in accuracy versus a 5% loss for log base-2 quantization for VGG-16 [117]. Furthermore, when weights are quantized to powers of two, the multiplication can be replaced with a bitshift [122,135].…”

Section: A Reduce Precisionmentioning

confidence: 99%

Efficient Processing of Deep Neural Networks: A Tutorial and Survey

et al. 2017

View full text Add to dashboard Cite

Abstract-Deep neural networks (DNNs) are currently widely used for many artificial intelligence (AI) applications including computer vision, speech recognition, and robotics. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Accordingly, techniques that enable efficient processing of DNNs to improve energy efficiency and throughput without sacrificing application accuracy or increasing hardware cost are critical to the wide deployment of DNNs in AI systems.This article aims to provide a comprehensive tutorial and survey about the recent advances towards the goal of enabling efficient processing of DNNs. Specifically, it will provide an overview of DNNs, discuss various hardware platforms and architectures that support DNNs, and highlight key trends in reducing the computation cost of DNNs either solely via hardware design changes or via joint hardware design and DNN algorithm changes. It will also summarize various development resources that enable researchers and practitioners to quickly get started in this field, and highlight important benchmarking metrics and design considerations that should be used for evaluating the rapidly growing number of DNN hardware designs, optionally including algorithmic co-designs, being proposed in academia and industry.The reader will take away the following concepts from this article: understand the key design considerations for DNNs; be able to evaluate different DNN hardware implementations with benchmarks and comparison metrics; understand the trade-offs between various hardware architectures and platforms; be able to evaluate the utility of various DNN design techniques for efficient processing; and understand recent implementation trends and opportunities.

show abstract

Section: A Reduce Precisionmentioning

confidence: 99%

Efficient Processing of Deep Neural Networks: A Tutorial and Survey

et al. 2017

View full text Add to dashboard Cite

show abstract

“…While logarithmic representation can also be used for activations, this has yet to be explored. LogNet's authors quantised CNNs with weights encoded in a four-bit logarithmic format, a er which they performed retraining to recover some lost accuracy [76]. eir experiments with the ImageNet dataset revealed 4.9 pp and 4.6 pp top-ve accuracy drops for AlexNet and VGG16, respectively.…”

Section: 3mentioning

confidence: 99%

“…In custom hardware, a multiplication between an exponentially quantised weight parameter and an activation can be implemented cheaply using a variable-length binary shi er. With LogNet, CNN inference is performed on FPGAs with four-bit logarithmic-quantised weights [76]. Experiments with three convolutional layers showed an over-3.0× energy e ciency improvement vs an Nvidia Titan X GPU implementation, while a four-bit logarithmic implementation of AlexNet demonstrated an around-5 pp accuracy loss for ImageNet.…”

Section: 32mentioning

confidence: 99%

Deep Neural Network Approximation for Custom Hardware

et al. 2019

View full text Add to dashboard Cite

LondonDeep neural networks have proven to be particularly e ective in visual and audio recognition tasks. Existing models tend to be computationally expensive and memory intensive, however, and so methods for hardwareoriented approximation have become a hot topic. Research has shown that custom hardware-based neural network accelerators can surpass their general-purpose processor equivalents in terms of both throughput and energy e ciency. Application-tailored accelerators, when co-designed with approximation-based network training methods, transform large, dense and computationally expensive networks into small, sparse and hardware-e cient alternatives, increasing the feasibility of network deployment. In this article, we provide a comprehensive evaluation of approximation methods for high-performance network inference along with in-depth discussion of their e ectiveness for custom hardware implementation. We also include proposals for future research based on a thorough analysis of current trends. is article represents the rst survey providing detailed comparisons of custom hardware accelerators featuring approximation for both convolutional and recurrent neural networks, through which we hope to inspire exciting new developments in the eld.

show abstract

“…Quantization, activation pruning are common practices in CNNs to reduce overheads associated with arithmetic operations. Different encoding based quantization schemes can be found in the literature, such as fixed-point linear quantization, logarithmic quantization, and binarization [ 33 , 34 , 35 ]. CNN accelerators supporting low precision can result in a resource-efficient solution.…”

Section: Introductionmentioning

confidence: 99%

Towards an Efficient CNN Inference Architecture Enabling In-Sensor Processing

Pantho

Bhowmik

Bobda

2021

Sensors

View full text Add to dashboard Cite

The astounding development of optical sensing imaging technology, coupled with the impressive improvements in machine learning algorithms, has increased our ability to understand and extract information from scenic events. In most cases, Convolution neural networks (CNNs) are largely adopted to infer knowledge due to their surprising success in automation, surveillance, and many other application domains. However, the convolution operations’ overwhelming computation demand has somewhat limited their use in remote sensing edge devices. In these platforms, real-time processing remains a challenging task due to the tight constraints on resources and power. Here, the transfer and processing of non-relevant image pixels act as a bottleneck on the entire system. It is possible to overcome this bottleneck by exploiting the high bandwidth available at the sensor interface by designing a CNN inference architecture near the sensor. This paper presents an attention-based pixel processing architecture to facilitate the CNN inference near the image sensor. We propose an efficient computation method to reduce the dynamic power by decreasing the overall computation of the convolution operations. The proposed method reduces redundancies by using a hierarchical optimization approach. The approach minimizes power consumption for convolution operations by exploiting the Spatio-temporal redundancies found in the incoming feature maps and performs computations only on selected regions based on their relevance score. The proposed design addresses problems related to the mapping of computations onto an array of processing elements (PEs) and introduces a suitable network structure for communication. The PEs are highly optimized to provide low latency and power for CNN applications. While designing the model, we exploit the concepts of biological vision systems to reduce computation and energy. We prototype the model in a Virtex UltraScale+ FPGA and implement it in Application Specific Integrated Circuit (ASIC) using the TSMC 90nm technology library. The results suggest that the proposed architecture significantly reduces dynamic power consumption and achieves high-speed up surpassing existing embedded processors’ computational capabilities.

show abstract

LogNet: Energy-efficient neural networks using logarithmic computation

Cited by 123 publications

References 6 publications

Efficient Processing of Deep Neural Networks: A Tutorial and Survey

Efficient Processing of Deep Neural Networks: A Tutorial and Survey

Deep Neural Network Approximation for Custom Hardware

Towards an Efficient CNN Inference Architecture Enabling In-Sensor Processing

Contact Info

Product

Resources

About