Approximating distributions in stochastic learning

Leen, Todd K.; Friel, Robert; Nielsen, David

doi:10.1016/j.neunet.2012.02.006

Cited by 4 publications

(4 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The model uses a residual convolutional neural network that enables the direct flow of information across layers by using skip connections which can improve the performance of deep learning models . Generally, having too few layers in a neural network may lead to underfitting where the model fails to capture the complexity of the data and performs poorly on both training and testing datasets, while having too many layers can cause overfitting, where the model becomes too specific to the training data and performs poorly on testing data. , We have optimized hyperparameters including epoch numbers, batch size, the initial learning rate, weight decay, numbers of layers, size of filters, initial input image size, and sensitivity to input channels both manually and with a grid search method. Based on our empirical tests covering hundreds of conventional possible combinations of parameters, we find that ten convolutional layers, max and average pooling layers, and one flatten layer are appropriate for our task.…”

Section: Data Sources and Methodsmentioning

confidence: 99%

Enhancing Global Estimation of Fine Particulate Matter Concentrations by Including Geophysical a Priori Information in Deep Learning

Shen,

Li,

van Donkelaar

et al. 2024

ACS EST Air

View full text Add to dashboard Cite

Global fine particulate matter (PM 2.5 ) assessment is impeded by a paucity of monitors. We improve estimation of the global distribution of PM 2.5 concentrations by developing, optimizing, and applying a convolutional neural network with information from satellite-, simulation-, and monitor-based sources to predict the local bias in monthly geophysical a priori PM 2.5 concentrations over 1998−2019. We develop a loss function that incorporates geophysical a priori estimates and apply it in model training to address the unrealistic results produced by meansquare-error loss functions in regions with few monitors. We introduce novel spatial cross-validation for air quality to examine the importance of considering spatial properties. We address the sharp decline in deep learning model performance in regions distant from monitors by incorporating the geophysical a priori PM 2.5 . The resultant monthly PM 2.5 estimates are highly consistent with spatial cross-validation PM 2.5 concentrations from monitors globally and regionally. We withheld 10% to 99% of monitors for testing to evaluate the sensitivity and robustness of model performance to the density of ground-based monitors. The model incorporating the geophysical a priori PM 2.5 concentrations remains highly consistent with observations globally even under extreme conditions (e.g., 1% for training, R 2 = 0.73), while the model without exhibits weaker performance (1% for training, R 2 = 0.51).

show abstract

Section: Data Sources and Methodsmentioning

confidence: 99%

Enhancing Global Estimation of Fine Particulate Matter Concentrations by Including Geophysical a Priori Information in Deep Learning

Shen,

Li,

van Donkelaar

et al. 2024

ACS EST Air

View full text Add to dashboard Cite

show abstract

“…CN network models examine conditions of theory-driven, kernel-based, or sparse-random connectivity, wherein sparsity is typically around 5-10% due to the breakdown of smaller (i.e., computationally feasible) models at more brain-like sparsity. Methods for scaling up CN models include partial inference such as isolated cell-type pre-tuning 35 , multi-region network of networks models 36 , and formal (e.g., mean field or master equation) approaches to mesoscopic dynamics 37,38 . Further, CN models typically study learning rules based on Hebbian association (or similar) and the types of local plasticity mechanisms that have been the focus of experiments.…”

Section: Obstacles For Network Models With Sparse Connectivitymentioning

confidence: 99%

A brain basis of dynamical intelligence for AI and computational neuroscience

Monaco¹,

Rajan²,

Hwang³

2021

Preprint

View full text Add to dashboard Cite

The deep neural nets of modern artificial intelligence (AI) have not achieved defining features of biological intelligence, including abstraction, causal learning, and energy-efficiency. While scaling to larger models has delivered performance improvements for current applications, more brain-like capacities may demand new theories, models, and methods for designing artificial learning systems. Here, we argue that this opportunity to reassess insights from the brain should stimulate cooperation between AI research and theory-driven computational neuroscience (CN). To motivate a brain basis of neural computation, we present a dynamical view of intelligence from which we elaborate concepts of sparsity in network structure, temporal dynamics, and interactive learning. In particular, we suggest that temporal dynamics, as expressed through neural synchrony, nested oscillations, and flexible sequences, provide a rich computational layer for reading and updating hierarchical models distributed in long-term memory networks. Moreover, embracing agent-centered paradigms in AI and CN will accelerate our understanding of the complex dynamics and behaviors that build useful world models. A convergence of AI/CN theories and objectives will reveal dynamical principles of intelligence for brains and engineered learning systems. This article was inspired by our symposium on dynamical neuroscience and machine learning at the 6th Annual US/NIH BRAIN Initiative Investigators Meeting. MainThe functional limitations of the current wave of artificial intelligence (AI), based on deploying deep neural nets for perception, language, and reinforcement learning applications, are coming into focus. A recent avalanche of reviews, perspectives, podcasts, virtual seminars, and keynotes have collectively signaled remarkable agreement about brain-like capacities that escape our understanding: abstraction, generalization, and compositionality; causal learning and inference; cognitive flexibility for generalized problem-solving; construction of world models; low sample-complexity and high energy-efficiency. These conversations have Monaco,

show abstract

“…Recent literatures have shown that the STDP based learning in SNN follow an Ornstein-Uhlenbeck process [28]. On the other hand, SGD follows a Feller process [29].…”

Section: B Generalizability Of Stdpmentioning

confidence: 99%

“…The Fokker-Planck (FP) equation tracks the resulting evolution of the SNN configurations θ over time, yielding the stationary distribution 1 z p * (θ) 1/T which follows the Ornstein-Uhlenbeck process [28] ∂ ∂t…”

Section: B Generalizability Of Stdpmentioning

confidence: 99%

A Fully Spiking Hybrid Neural Network for Energy-Efficient Object Detection

Chakraborty,

She,

Mukhopadhyay

2021

Preprint

View full text Add to dashboard Cite

This paper proposes a Fully Spiking Hybrid Neural Network (FSHNN) for energy-efficient and robust object detection in resource-constrained platforms. The network architecture is based on Convolutional SNN using leaky-integrate-fire neuron models. The model combines unsupervised Spike Time-Dependent Plasticity (STDP) learning with back-propagation (STBP) learning methods and also uses Monte Carlo Dropout to get an estimate of the uncertainty error. FSHNN provides better accuracy compared to DNN based object detectors while being 150X energy-efficient. It also outperforms these object detectors, when subjected to noisy input data and less labeled training data with a lower uncertainty error.

show abstract

Approximating distributions in stochastic learning

Cited by 4 publications

References 30 publications

Enhancing Global Estimation of Fine Particulate Matter Concentrations by Including Geophysical a Priori Information in Deep Learning

Enhancing Global Estimation of Fine Particulate Matter Concentrations by Including Geophysical a Priori Information in Deep Learning

A brain basis of dynamical intelligence for AI and computational neuroscience

A Fully Spiking Hybrid Neural Network for Energy-Efficient Object Detection

Contact Info

Product

Resources

About