Through the success of deep learning in various domains, artificial neural networks are currently among the most used artificial intelligence methods. Taking inspiration from the network properties of biological neural networks (e.g. sparsity, scale-freeness), we argue that (contrary to general practice) artificial neural networks, too, should not have fully-connected layers. Here we propose sparse evolutionary training of artificial neural networks, an algorithm which evolves an initial sparse topology (Erdős–Rényi random graph) of two consecutive layers of neurons into a scale-free topology, during learning. Our method replaces artificial neural networks fully-connected layers with sparse ones before training, reducing quadratically the number of parameters, with no decrease in accuracy. We demonstrate our claims on restricted Boltzmann machines, multi-layer perceptrons, and convolutional neural networks for unsupervised and supervised learning on 15 datasets. Our approach has the potential to enable artificial neural networks to scale up beyond what is currently possible.
Unprecedented high volumes of data are becoming available with the growth of the advanced metering infrastructure. These are expected to benefit planning and operation of the future power system, and to help the customers transition from a passive to an active role. In this paper, we explore for the first time in the smart grid context the benefits of using Deep Reinforcement Learning, a hybrid type of methods that combines Reinforcement Learning with Deep Learning, to perform on-line optimization of schedules for building energy management systems. The learning procedure was explored using two methods, Deep Q-learning and Deep Policy Gradient, both of them being extended to perform multiple actions simultaneously. The proposed approach was validated on the large-scale Pecan Street Inc. database. This highly-dimensional database includes information about photovoltaic power generation, electric vehicles as well as buildings appliances. Moreover, these on-line energy scheduling strategies could be used to provide realtime feedback to consumers to encourage more efficient use of electricity.
Restricted Boltzmann Machines (RBMs) and models derived from them have been
successfully used as basic building blocks in deep artificial neural networks
for automatic features extraction, unsupervised weights initialization, but
also as density estimators. Thus, their generative and discriminative
capabilities, but also their computational time are instrumental to a wide
range of applications. Our main contribution is to look at RBMs from a
topological perspective, bringing insights from network science. Firstly, here
we show that RBMs and Gaussian RBMs (GRBMs) are bipartite graphs which
naturally have a small-world topology. Secondly, we demonstrate both on
synthetic and real-world datasets that by constraining RBMs and GRBMs to a
scale-free topology (while still considering local neighborhoods and data
distribution), we reduce the number of weights that need to be computed by a
few orders of magnitude, at virtually no loss in generative performance.
Thirdly, we show that, for a fixed number of weights, our proposed sparse
models (which by design have a higher number of hidden neurons) achieve better
generative capabilities than standard fully connected RBMs and GRBMs (which by
design have a smaller number of hidden neurons), at no additional computational
costs.Comment: http://link.springer.com/article/10.1007/s10994-016-5570-z, Machine
Learning, issn=1573-0565, 201
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.