The Stochastic Delta Rule: Faster and More Accurate Deep Learning Through Adaptive Weight Noise

Frazier-Logue, Noah; Hanson, Stephen José

doi:10.1162/neco_a_01276

Cited by 11 publications

(4 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We note that in machine learning, noise processes such as dropout and stochastic regularization (e.g., refs. 30–32) can be applied to improve generalization from finite training data. Intrinsic synaptic noise is qualitatively different from these regularization processes.…”

Section: Resultsmentioning

confidence: 99%

Fundamental bounds on learning performance in neural circuits

Raman

Rotondo

O’Leary

2019

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

How does the size of a neural circuit influence its learning performance? Larger brains tend to be found in species with higher cognitive function and learning ability. Intuitively, we expect the learning capacity of a neural circuit to grow with the number of neurons and synapses. We show how adding apparently redundant neurons and connections to a network can make a task more learnable. Consequently, large neural circuits can either devote connectivity to generating complex behaviors or exploit this connectivity to achieve faster and more precise learning of simpler behaviors. However, we show that in a biologically relevant setting where synapses introduce an unavoidable amount of noise, there is an optimal size of network for a given task. Above the optimal network size, the addition of neurons and synaptic connections starts to impede learning performance. This suggests that the size of brain circuits may be constrained by the need to learn efficiently with unreliable synapses and provides a hypothesis for why some neurological learning deficits are associated with hyperconnectivity. Our analysis is independent of specific learning rules and uncovers fundamental relationships between learning rate, task performance, network size, and intrinsic noise in neural circuits.

show abstract

Section: Resultsmentioning

confidence: 99%

Fundamental bounds on learning performance in neural circuits

Raman

Rotondo

O’Leary

2019

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…In the past, various CNN architectures and modifications have been developed, and some show a high classification accuracy in the ImageNet dataset [ 49 , 50 , 51 ]. However, as the newer CNN architectures were not (yet) implemented in the software that was used here, we choose CNN architectures that were previously used to classify image data and were available in our software.…”

Section: Discussionmentioning

confidence: 99%

Deep Learning for the Classification of Small-Cell and Non-Small-Cell Lung Cancer

Kriegsmann

Haag

Weis

et al. 2020

Cancers

View full text Add to dashboard Cite

Reliable entity subtyping is paramount for therapy stratification in lung cancer. Morphological evaluation remains the basis for entity subtyping and directs the application of additional methods such as immunohistochemistry (IHC). The decision of whether to perform IHC for subtyping is subjective, and access to IHC is not available worldwide. Thus, the application of additional methods to support morphological entity subtyping is desirable. Therefore, the ability of convolutional neuronal networks (CNNs) to classify the most common lung cancer subtypes, pulmonary adenocarcinoma (ADC), pulmonary squamous cell carcinoma (SqCC), and small-cell lung cancer (SCLC), was evaluated. A cohort of 80 ADC, 80 SqCC, 80 SCLC, and 30 skeletal muscle specimens was assembled; slides were scanned; tumor areas were annotated; image patches were extracted; and cases were randomly assigned to a training, validation or test set. Multiple CNN architectures (VGG16, InceptionV3, and InceptionResNetV2) were trained and optimized to classify the four entities. A quality control (QC) metric was established. An optimized InceptionV3 CNN architecture yielded the highest classification accuracy and was used for the classification of the test set. Image patch and patient-based CNN classification results were 95% and 100% in the test set after the application of strict QC. Misclassified cases mainly included ADC and SqCC. The QC metric identified cases that needed further IHC for definite entity subtyping. The study highlights the potential and limitations of CNN image classification models for tumor differentiation.

show abstract

“…In order to validate the denoising effectiveness of the algorithms proposed in this article, several traditional and literature algorithms are selected for comparison ( Chen & Han, 2005 ; Hu et al, 2021 ; Deng et al, 2020 ; Frazier-Logue & José Hanson, 2020 ). The simulation process involves collecting data for each algorithm and comparing them.…”

Section: Simulationmentioning

confidence: 99%

A 2D image 3D reconstruction function adaptive denoising algorithm

Wang,

Ni,

Liu

et al. 2023

PeerJ Computer Science

View full text Add to dashboard Cite

To address the issue of image denoising algorithms blurring image details during the denoising process, we propose an adaptive denoising algorithm for the 3D reconstruction of 2D images. This algorithm takes into account the inherent visual characteristics of human eyes and divides the image into regions based on the entropy value of each region. The background region is subject to threshold denoising, while the target region undergoes processing using an adversarial generative network. This network effectively handles 2D target images with noise and generates a 3D model of the target. The proposed algorithm aims to enhance the noise immunity of 2D images during the 3D reconstruction process and ensure that the constructed 3D target model better preserves the original image’s detailed information. Through experimental testing on 2D images and real pedestrian videos contaminated with noise, our algorithm demonstrates stable preservation of image details. The reconstruction effect is evaluated in terms of noise reduction and the fidelity of the 3D model to the original target. The results show an average noise reduction exceeding 95% while effectively retaining most of the target’s feature information in the original image. In summary, our proposed adaptive denoising algorithm improves the 3D reconstruction process by preserving image details that are often compromised by conventional denoising techniques. This has significant implications for enhancing image quality and maintaining target information fidelity in 3D models, providing a promising approach for addressing the challenges associated with noise reduction in 2D images during 3D reconstruction.

show abstract

The Stochastic Delta Rule: Faster and More Accurate Deep Learning Through Adaptive Weight Noise

Cited by 11 publications

References 6 publications

Fundamental bounds on learning performance in neural circuits

Fundamental bounds on learning performance in neural circuits

Deep Learning for the Classification of Small-Cell and Non-Small-Cell Lung Cancer

A 2D image 3D reconstruction function adaptive denoising algorithm

Contact Info

Product

Resources

About