Effect of injected noise in deep neural networks

Nagabushan, Naresh; Satish, N.; S, Raghuram

doi:10.1109/iccic.2016.7919546

Cited by 8 publications

(4 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We attribute this to the robustness of the nonlinear network to the experimental noise. There is a large body of ongoing research in the machine learning community on the effect of noise in training deep neural network [17][18][19], and the exact nature of the robustness of our nonlinear optical neural network remains to be investigated. Table 1 summarizes all the accuracy results for both simulation and experiment.…”

Section: Resultsmentioning

confidence: 99%

Free-space optical neural network based on thermal atomic nonlinearity

et al. 2021

View full text Add to dashboard Cite

As artificial neural networks (ANNs) continue to make strides in wide-ranging and diverse fields of technology, the search for more efficient hardware implementations beyond conventional electronics is gaining traction. In particular, optical implementations potentially offer extraordinary gains in terms of speed and reduced energy consumption due to intrinsic parallelism of free-space optics. At the same time, a physical nonlinearity -a crucial ingredient of an ANN -is not easy to realize in free-space optics, which restricts the potential of this platform. This problem is further exacerbated by the need to perform the nonlinear activation also in parallel for each data point to preserve the benefit of linear free-space optics. Here, we present a free-space optical ANN with diffraction-based linear weight summation and nonlinear activation enabled by the saturable absorption of thermal atoms. We demonstrate, via both simulation and experiment, image classification of handwritten digits using only a single layer and observed 6% improvement in classification accuracy due to the optical nonlinearity compared to a linear model. Our platform preserves the massive parallelism of free-space optics even with physical nonlinearity, and thus opens the way for novel designs and wider deployment of optical ANNs.

show abstract

Section: Resultsmentioning

confidence: 99%

Free-space optical neural network based on thermal atomic nonlinearity

et al. 2021

View full text Add to dashboard Cite

show abstract

“…This leads to poor performance when a new input was never seen before. Some techniques that help to avoid overfitting by making the model simpler are dropout [115], early stopping [110], weight decay [70] and learning with noise [83]. As discussed in Section 3, the approximation noise induced by the approximate circuits might help in mitigating overfitting.…”

Section: Trainingmentioning

confidence: 99%

Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey

et al. 2022

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) are very popular because of their high performance in various cognitive tasks in Machine Learning (ML). Recent advancements in DNNs have brought beyond human accuracy in many tasks, but at the cost of high computational complexity. To enable efficient execution of DNN inference, more and more research works, therefore, exploit the inherent error resilience of DNNs and employ Approximate Computing (AC) principles to address the elevated energy demands of DNN accelerators. This article provides a comprehensive survey and analysis of hardware approximation techniques for DNN accelerators. First, we analyze the state of the art and by identifying approximation families, we cluster the respective works with respect to the approximation type. Next, we analyze the complexity of the performed evaluations (with respect to the dataset and DNN size) to assess the efficiency, the potential, and limitations of approximate DNN accelerators. Moreover, a broad discussion is provided, regarding error metrics that are more suitable for designing approximate units for DNN accelerators as well as accuracy recovery approaches that are tailored to DNN inference. Finally, we present how Approximate Computing for DNN accelerators can go beyond energy efficiency and address reliability and security issues, as well.

show abstract

“…The SNR is defined as the ratio of the average of the output of the activation function divided by the variance [10]. Considering noise in the process of training has been shown to improve resilience towards the ubiquitous noise and performance [8,15,16]. Furthermore, in any implementation of the NN noise is present, therefore a more realistic assumption is to include it in the training stage as well.…”

Section: Crowd Equalisationmentioning

confidence: 99%