Hardware-Aware Softmax Approximation for Deep Neural Networks

Geng, Xue; Lin, Jie; Zhao, Bin; Kong, Adams Wai-Kin; Aly, Mohamed M. Sabry; Chandrasekhar, Vijay

doi:10.1007/978-3-030-20870-7_7

Cited by 21 publications

(13 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, use of CNN is hindered in small hardwares, especially in mobile devices because of its high computational cost. In this regard, different hardware accelerators are needed for reducing both execution time and power consumption [247]. Some of the very interesting accelerators are already proposed.…”

Section: Future Directionsmentioning

confidence: 99%

A survey of the recent architectures of deep convolutional neural networks

et al. 2020

View full text Add to dashboard Cite

Deep Convolutional Neural Networks (CNNs) are a special type of Neural Networks, which have shown exemplary performance on several competitions related to Computer Vision and Image Processing. Interesting application areas of CNN include Image Classification and Segmentation, Object Detection, Video Processing, Natural Language Processing, Speech Recognition, etc. The powerful learning ability of deep CNN is largely due to the use of multiple feature extraction stages that can automatically learn representations from the data. Availability of a large amount of data and improvements in the hardware technology have accelerated the research in CNNs, and recently very interesting deep CNN architectures have been reported. In fact, several interesting ideas to bring advancements in CNNs have been explored such as the use of different activation and loss functions, parameter optimization, regularization, and architectural innovations. However, the major improvement in representational capacity of the deep CNN is achieved through architectural innovations. Especially, the idea of exploiting spatial and channel information, depth and width of architecture, and multi-path information processing has gained substantial attention. Similarly, the idea of using a block of layers as a structural unit is also gaining popularity. This survey thus focuses on the intrinsic taxonomy present in the recently reported deep CNN architectures and consequently, classifies the recent innovations in CNN architectures into seven different categories. These seven categories are based on spatial exploitation, depth, multi-path, width, feature-map exploitation, channel boosting, and attention. Additionally, the elementary understanding of CNN components, current challenges and applications of CNN are also provided. CNNs are the best among learning algorithms in understanding images content, and have shown exemplary results in segmentation, classification, detection, and retrieval related tasks [8], [9]. The success of CNNs has captured attention beyond academia. In industry, companies such as Google, Microsoft, AT&T, NEC, and Facebook have developed active research groups for exploring new architectures of CNN [10]. At present, most of the frontrunners of image processing and computer vision competitions are employing deep CNN based models.The attractive feature of CNN is its ability to exploit spatial or time correlation of the data. The topology of CNN is divided into multiple learning stages composed of a combination of the convolutional layers, non-linear processing units, and subsampling layers [11]. CNNs are feedforward multilayered hierarchical networks that are similar to fully connected neural network where each layer, using a bank of convolutional kernels, performs multiple transformations [12]. Convolution operation extracts useful features from locally correlated data points. Output of the convolutional kernels is assigned to non-linear processing unit (activation function), which not only helps in learning abstractions but also emb...

show abstract

Section: Future Directionsmentioning

confidence: 99%

A survey of the recent architectures of deep convolutional neural networks

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Conversely, the softmax operations used in the attention module are approximated following a more complex methodology, similar to the one used in [26]. Consider the matrix M as defined in Equation 1 and a given row j in M , and a vector m j as input to the softmax operation.…”

Section: Integer Precision Approximationmentioning

confidence: 99%

Attention-Based Neural Networks for Chroma Intra Prediction in Video Coding

Blanch¹,

Blasi²,

Smeaton

et al. 2021

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

Neural networks can be successfully used to improve several modules of advanced video coding schemes. In particular, compression of colour components was shown to greatly benefit from usage of machine learning models, thanks to the design of appropriate attention-based architectures that allow the prediction to exploit specific samples in the reference region. However, such architectures tend to be complex and computationally intense, and may be difficult to deploy in a practical video coding pipeline. This work focuses on reducing the complexity of such methodologies, to design a set of simplified and cost-effective attention-based architectures for chroma intra-prediction. A novel size-agnostic multi-model approach is proposed to reduce the complexity of the inference process. The resulting simplified architecture is still capable of outperforming state-of-the-art methods. Moreover, a collection of simplifications is presented in this paper, to further reduce the complexity overhead of the proposed prediction architecture. Thanks to these simplifications, a reduction in the number of parameters of around 90% is achieved with respect to the original attentionbased methodologies. Simplifications include a framework for reducing the overhead of the convolutional operations, a simplified cross-component processing model integrated into the original architecture, and a methodology to perform integer-precision approximations with the aim to obtain fast and hardware-aware implementations. The proposed schemes are integrated into the Versatile Video Coding (VVC) prediction pipeline, retaining compression efficiency of state-of-the-art chroma intra-prediction methods based on neural networks, while offering different directions for significantly reducing coding complexity.

show abstract

“…But there is still problem when the CNN architectures are consuming large resources for computation purpose and also overhead [26][27][28]. Various types of hardware accelerators and their architectures are discussed in [29] for reducing the power consumption and large overheads. An example of such accelerator is FGPA discussed in [30] for minimizing power consumption.…”

Section: Literature Reviewmentioning

confidence: 99%

“…An example of such accelerator is FGPA discussed in [30] for minimizing power consumption. Further, using the modern techniques, some hardware related models were introduced in [31]. The modern research also relies on the various optimization techniques.…”

Section: Literature Reviewmentioning

confidence: 99%

“…By using these, three parameters, we can easily reduce the size and dimensionality of the parameters of convolutional layers output. We can use this formula for applying these parameters given by Keiron O' Shea [31] (I− R) + 2P/(S+1) 2. S + Where I represent the input size, R is the receptive field size, P is the amount of adding zero padding and S is the stride.…”

Section: Functions Of Cnn Layersmentioning

confidence: 99%

See 1 more Smart Citation

Pneumonia Classification using Deep Learning in Healthcare

Verma¹,

Prakash²

2020

IJITEE

View full text Add to dashboard Cite

There is a great growing interest in the domain of deep learning techniques for identifying and classifying images with various datasets. An enormous availability of datasets (e.g. ChestX-Ray14 dataset) has developed a keen interest in deep learning. Pneumonia is a disease that is caused by various bacteria, virus etc. X-ray is one of the major diagnosis tools for diagnosing pneumonia. This research work mainly proposes a convolutional neural system (CNN) model prepared without any preparation to group and identify the occurrence of pneumonia disease from a given assortment of chest X-ray image tests. Dissimilar to different strategies that depend exclusively on more learning draws near or conventional carefully assembled systems to accomplish an amazing grouping execution, and developed a convolutional neural arrange model without any preparation to separate and character the images to decide whether an individual is suffering with pneumonia. This model could help alleviate the dependability and difficult challenges frequently confronted to manage therapeutic problems. In this paper, CNN algorithm has been used along with different data augmentation techniques for improving the classification accuracies which has been discussed to increase the performance which will help in improving the validation and training accuracies and characterization of exactness of the CNN model and accomplished various results. This experiment was carried out using python language and has shown improved outcomes.

show abstract

Hardware-Aware Softmax Approximation for Deep Neural Networks

Cited by 21 publications

References 15 publications

A survey of the recent architectures of deep convolutional neural networks

A survey of the recent architectures of deep convolutional neural networks

Attention-Based Neural Networks for Chroma Intra Prediction in Video Coding

Pneumonia Classification using Deep Learning in Healthcare

Contact Info

Product

Resources

About