AX-DBN: An Approximate Computing Framework for the Design of Low-Power Discriminative Deep Belief Networks

Colbert, Ian; Kreutz-Delgado, Kenneth; Das, Srinjoy

doi:10.1109/ijcnn.2019.8852476

Cited by 3 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The concept of using an iterative pruning schedule to alleviate the impact of abruptly removing neurons has been explored in prior work [11,23,37,38]; however, they iteratively introduce sparsity globally at each step. Because these pruning schedules gradually introduce sparsity according to a per-step heuristic, we refer to this class of algorithms as "stepwise" pruning.…”

Section: Inputmentioning

confidence: 99%

“…Pruning (i.e., the process of removing identified redundant elements from a neural network) and quantization (i.e., the processing of reducing their precision) are often considered to be independent problems [7,8]; however, recent work has begun to study the application of both in either a joint [2,6,9,10] or unified [11,12] setting. Unified algorithms typically use mixed precision quantization and integrate pruning by reducing the precision of an element (or a set of elements) to 0.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning Low-Precision Structured Subnetworks Using Joint Layerwise Channel Pruning and Uniform Quantization

2022

View full text Add to dashboard Cite

Pruning and quantization are core techniques used to reduce the inference costs of deep neural networks. Among the state-of-the-art pruning techniques, magnitude-based pruning algorithms have demonstrated consistent success in the reduction of both weight and feature map complexity. However, we find that existing measures of neuron (or channel) importance estimation used for such pruning procedures have at least one of two limitations: (1) failure to consider the interdependence between successive layers; and/or (2) performing the estimation in a parametric setting or by using distributional assumptions on the feature maps. In this work, we demonstrate that the importance rankings of the output neurons of a given layer strongly depend on the sparsity level of the preceding layer, and therefore, naïvely estimating neuron importance to drive magnitude-based pruning will lead to suboptimal performance. Informed by this observation, we propose a purely data-driven nonparametric, magnitude-based channel pruning strategy that works in a greedy manner based on the activations of the previous sparsified layer. We demonstrate that our proposed method works effectively in combination with statistics-based quantization techniques to generate low precision structured subnetworks that can be efficiently accelerated by hardware platforms such as GPUs and FPGAs. Using our proposed algorithms, we demonstrate increased performance per memory footprint over existing solutions across a range of discriminative and generative networks.

show abstract

Section: Inputmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Learning Low-Precision Structured Subnetworks Using Joint Layerwise Channel Pruning and Uniform Quantization

2022

View full text Add to dashboard Cite

show abstract

“…In an effort to further reduce processing requirements, some RFML implementations have also embedded traditional signal processing techniques such as Fourier and wavelet transforms, cyclostationary feature estimators, and other expert features directly into the NN [170], [174], [175]. Meanwhile, other research has focused on reduced precision implementations of NNs, enabling a path towards real-time implementation [176]- [178]. However, reducing real-time computational resources to mobile systems remains a challenge that must be overcome, especially if online learning techniques are to be developed for future RFML systems [179], [180].…”

Section: A Size Weight and Power (Swap)mentioning

confidence: 99%

An RFML Ecosystem: Considerations for the Application of Deep Learning to Spectrum Situational Awareness

Wong

Clark

Flowers

et al. 2021

IEEE Open J. Commun. Soc.

View full text Add to dashboard Cite

While deep learning (DL) technologies are now pervasive in state-of-the-art Computer Vision (CV) and Natural Language Processing (NLP) applications, only in recent years have these technologies started to sufficiently mature in applications related to wireless communications, a field loosely termed Radio Frequency Machine Learning (RFML). In particular, recent research has shown DL to be an enabling technology for Cognitive Radio (CR) applications as well as a useful tool for supplementing expertly defined algorithms for spectrum awareness applications such as signal detection, estimation, and classification. A major driver for the usage of RFML is that little, to no, a priori knowledge of the intended spectral environment is required, given that there is an abundance of representative raw Radio Frequency (RF) data to facilitate training and evaluation. However, in addition to this fundamental need for sufficient data, there are other key considerations, such as trust, security, and hardware requirements, that must be taken into account before deploying RFML systems in real-world wireless communication applications that largely go unaddressed in the current literature. This paper examines the prior works related to these major research considerations, with focus on the dependencies between them and factors unique to the RFML space.

show abstract

“…Further, some RFML implementations incorporate pre-calculated traditional signal processing techniques such as Fourier and wavelet transforms, cyclostationary feature estimators, and other expert features to serve as a more efficient feature that may be merged with machine learned behaviors [241], [245], [246]. Other research has focused on reduced precision implementations of machine learning structures as a method to gain computational efficiency [247]- [249]. However, the use of online learning techniques in RF scenarios requires real-time computational resources that are currently difficult to reduce to a mobile system [250], [251], in addition to the challenges discussed in Section III.…”

Section: Deploymentmentioning

confidence: 99%

The RFML Ecosystem: A Look at the Unique Challenges of Applying Deep Learning to Radio Frequency Applications

Wong,

Clark,

Flowers

et al. 2020

Preprint

View full text Add to dashboard Cite

While deep machine learning technologies are now pervasive in state-of-the-art image recognition and natural language processing applications, only in recent years have these technologies started to sufficiently mature in applications related to wireless communications. In particular, recent research has shown deep machine learning to be an enabling technology for cognitive radio applications as well as a useful tool for supplementing expertly defined algorithms for spectrum sensing applications such as signal detection, estimation, and classification (termed here as Radio Frequency Machine Learning, or RFML). A major driver for the usage of deep machine learning in the context of wireless communications is that little, to no, a priori knowledge of the intended spectral environment is required, given that there is an abundance of representative data to facilitate training and evaluation. However, in addition to this fundamental need for sufficient data, there are other key considerations, such as trust, security, and hardware/software issues, that must be taken into account before deploying deep machine learning systems in real-world wireless communication applications. This paper provides an overview and survey of prior work related to these major research considerations. In particular, we present their unique considerations in the RFML application space, which are not generally present in the image, audio, and/or text application spaces.

show abstract

AX-DBN: An Approximate Computing Framework for the Design of Low-Power Discriminative Deep Belief Networks

Cited by 3 publications

References 15 publications

Learning Low-Precision Structured Subnetworks Using Joint Layerwise Channel Pruning and Uniform Quantization

Learning Low-Precision Structured Subnetworks Using Joint Layerwise Channel Pruning and Uniform Quantization

An RFML Ecosystem: Considerations for the Application of Deep Learning to Spectrum Situational Awareness

The RFML Ecosystem: A Look at the Unique Challenges of Applying Deep Learning to Radio Frequency Applications

Contact Info

Product

Resources

About