DensePILAE: a feature reuse pseudoinverse learning algorithm for deep stacked autoencoder

Abstract

Autoencoder has been widely used as a feature learning technique. In many works of autoencoder, the features of the original input are usually extracted layer by layer using multi-layer nonlinear mapping, and only the features of the last layer are used for classification or regression. Therefore, the features of the previous layer aren’t used explicitly. The loss of information and waste of computation is obvious. In addition, faster training and reasoning speed is generally required in the Internet of Things applications. But the stacked autoencoders model is usually trained by the BP algorithm, which has the problem of slow convergence. To solve the above two problems, the paper proposes a dense connection pseudoinverse learning autoencoder (DensePILAE) from reuse perspective. Pseudoinverse learning autoencoder (PILAE) can extract features in the form of analytic solution, without multiple iterations. Therefore, the time cost can be greatly reduced. At the same time, the features of all the previous layers in stacked PILAE are combined as the input of next layer. In this way, the information of all the previous layers not only has no loss, but also can be strengthened and refined, so that better features could be learned. The experimental results in 8 data sets of different domains show that the proposed DensePILAE is effective.

ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection

“…The multilayer analytic learning [23], [24] converts the nonlinear network learning into linear segments that can be solved adopting LS techniques in a oneepoch training style. For instance, the dense pseudoinverse autoencoder [25] trains a stacked autoencoder layer-by-layer by concatenating shallow and deep features using LS solutions. The analytic learning could experience out-of-memory issue as the weights are learned involving the entire dataset at once.…”