Training Multilayer Neural Networks Analytically Using Kernel Projection

Abstract

This paper proposes a kernel projection (KP) neural network that analytically determines its network parameters. The proposed network is composed of cascaded modules of 2-layer sub-networks. A technique which encodes the label information into each module has been introduced to enable a locally supervised learning. Such a supervised learning in the 2-layer module begins with a kernel projection in the first layer and determines its parameters analytically via solving a least squares problem in the second layer. We show that the analytic nature of the proposed network allows a learning process significantly faster than that of the traditional backpropagation method as it only needs to visit the dataset once. Experiments of classification tasks on various datasets are carried out, showing comparable or better results compared with several competing methods.Index Terms-analytic learning, least squares, kernel project.

Learning with Local Gradients at the Edge

“…In target projection [15,12], one hot encodings of the labels are directly projected to a given layer during the optimization step. Given an intermediate layer L i in a NN, we generate local targets y i for the layer by projecting the data labels y * via a random projection matrix (P i ).…”