Online Deep Learning: Learning Deep Neural Networks on the Fly

Abstract

Deep Neural Networks (DNNs) are typically trained by backpropagation in a batch learning setting, which requires the entire training data to be made available prior to the learning task. This is not scalable for many real-world scenarios where new data arrives sequentially in a stream form. We aim to address an open challenge of "Online Deep Learning" (ODL) for learning DNNs on the fly in an online setting. Unlike traditional online learning that often optimizes some convex objective function with respect to a shallow model (e.g., a linear/kernel-based hypothesis), ODL is significantly more challenging since the optimization of the DNN objective function is non-convex, and regular backpropagation does not work well in practice, especially for online learning settings. In this paper, we present a new online deep learning framework that attempts to tackle the challenges by learning DNN models of adaptive depth from a sequence of training data in an online learning setting. In particular, we propose a novel Hedge Backpropagation (HBP) method for online updating the parameters of DNN effectively, and validate the efficacy of our method on large-scale data sets, including both stationary and concept drifting scenarios.

DEVDAN: Deep evolving denoising autoencoder

“…The structural learning scenario is mainly driven by feature similarity and does not fully operate in the one-pass learning mode. [12] puts forward the hedge backpropagation method to answer the research question as to how and when a DNN structure should be adapted. This work, however, assumes that an initial structure of DNN exists and is built upon a fixed-capacity network.…”

Automatic Construction of Multi-layer Perceptron Network from Streaming Examples

“…The key difference lies in the calculation of mean and variance directly obtained from the bias itself rather than from the binomial distribution because the hidden unit growing strategy analyzes a real variablebias instead of the accuracy score. The high bias problem leading to the introduction of new hidden unit is formulated as follows: µ t bias + σ t bias ≥ µ min bias + πσ min bias (4) where π = 1.25exp(−Bias 2 ) + 0.75 and controls the confidence degree of sigma rule. It is observed that π is set adaptive as a factor of Bias and revolves around [1,2].…”

“…µ t var + σ t var ≥ µ min var + 2χσ min var (5) Compared to (4), the term 2 is introduced and meant to avoid a direct pruning after adding situation since addition of a new hidden unit leads to temporary increase of network variance but gradually decreases as next observations are come across. χ is set akin to π in the (4) as π = 1.25exp(−V ariance 2 ) + 0.75 which consequently causes k sigma rule in the range of [1,4]. This strategy navigates to between 68.2% and 99.9% confidence level.…”

“…[5] is crafted under a shallow network structure and the structural learning mechanism is done with the absence of particular criterion. Furthermore, ODL [4] does not explore the issue of structural learning and simply works with a predefined network structure.…”

“…The elastic consolidation weight (ECW) method is proposed in [1] where it overcomes the catastrophic forgetting issue by preventing a large deviation of output weights from the old one. Online deep learning (ODL) is proposed in [4] which introduces the idea of hedging. The hedging concept opens a direct connection of hidden layer to output layer with a connective weight and shares some similarity to the concept of weighted voting scheme in the ensemble learning literature.…”