Kafnets: Kernel-based non-parametric activation functions for neural networks

Scardapane, Simone; Vaerenbergh, Steven Van; Totaro, Simone; Uncini, Aurelio

doi:10.1016/j.neunet.2018.11.002

Cited by 62 publications

(77 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also linear networks (i.e., when φ is the identity) are commonly studied, both for the proven impact on applications and the very interesting results that can be derived in closed-form 24,25,27,31,32 . Other activation functions have been proposed in the neural networks literature, including those that normalize the activation values on a closed hyper-surface 33 and those based on non-parametric estimation via composition of kernel functions 34 . The output y ∈ R N out is generated by the matrix W out , whose weights are learned: generally by ridge regression or lasso 3, 35 but also with online training mechanisms 36 .…”

Section: Echo State Networkmentioning

confidence: 99%

Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere

Verzelli

Alippi

Livi

2019

Sci Rep

View full text Add to dashboard Cite

Among the various architectures of Recurrent Neural Networks, Echo State Networks (ESNs) emerged due to their simplified and inexpensive training procedure. These networks are known to be sensitive to the setting of hyper-parameters, which critically affect their behavior. Results show that their performance is usually maximized in a narrow region of hyper-parameter space called edge of criticality. Finding such a region requires searching in hyper-parameter space in a sensible way: hyper-parameter configurations marginally outside such a region might yield networks exhibiting fully developed chaos, hence producing unreliable computations. The performance gain due to optimizing hyper-parameters can be studied by considering the memory–nonlinearity trade-off, i.e., the fact that increasing the nonlinear behavior of the network degrades its ability to remember past inputs, and vice-versa. In this paper, we propose a model of ESNs that eliminates critical dependence on hyper-parameters, resulting in networks that provably cannot enter a chaotic regime and, at the same time, denotes nonlinear behavior in phase space characterized by a large memory of past inputs, comparable to the one of linear networks. Our contribution is supported by experiments corroborating our theoretical findings, showing that the proposed model displays dynamics that are rich-enough to approximate many common nonlinear systems used for benchmarking.

show abstract

Section: Echo State Networkmentioning

confidence: 99%

Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere

Verzelli

Alippi

Livi

2019

Sci Rep

View full text Add to dashboard Cite

show abstract

“…As stated in the introduction, as a further contribution we also decided to compare the standard NNs to the Kafnets [26], a class of neural networks using KAFs as activation functions. Briefly, using this architecture each activation function is allowed to change shape during training using a small number of adaptable coefficients, and the aim of this set of comparisons is to explore whether this additional number of degrees of freedom is beneficial or not in terms of CL.…”

Section: Architectures and Methodsmentioning

confidence: 99%

“…Each Kafnet used has the same architecture of the NN counterpart but the size of each layer/kernel is reduced by 30%, in order to have roughly the same number of adaptable parameters for both architectures. The hyper-parameters for the KAFs use the same values from [26].…”

Section: Architectures and Methodsmentioning

confidence: 99%

“…Because our method requires a small external memory of past activations to work efficiently, we also devise a simple sampling strategy for this memory to maximize the information provided by the regularization term, i.e., we try to sample maximally those elements in memory that can provide the strongest regularization on the model. In addition to the previous contributions, we also evaluate the impact of more flexible activation functions, namely, kernel activation functions (KAFs) [26], on the CL scenario. Several papers recently have shown that, in the single-task scenario, training the activation functions [8,1,11,22,26] can provide significant improvements in performance and in the number of layers required to solve the given task.…”

Section: Contributions Of This Papermentioning

confidence: 99%

See 1 more Smart Citation

Efficient continual learning in neural networks with embedding regularization

et al. 2020

Self Cite

View full text Add to dashboard Cite

Continual learning of deep neural networks is a key requirement for scaling them up to more complex applicative scenarios and for achieving real lifelong learning of these architectures. Previous approaches to the problem have considered either the progressive increase in the size of the networks, or have tried to regularize the network behavior to equalize it with respect to previously observed tasks. In the latter case, it is essential to understand what type of information best represents this past behavior. Common techniques include regularizing the past outputs, gradients, or individual weights. In this work, we propose a new, relatively simple and efficient method to perform continual learning by regularizing instead the network internal embeddings. To make the approach scalable, we also propose a dynamic sampling strategy to reduce the memory footprint of the required external storage. We show that our method performs favorably with respect to state-of-the-art approaches in * Corresponding author. Phone: +39 06 44585495.the literature, while requiring significantly less space in memory and computational time. In addition, inspired inspired by to recent works, we evaluate the impact of selecting a more flexible model for the activation functions inside the network, evaluating the impact of catastrophic forgetting on the activation functions themselves.

show abstract

“…None of the known approaches possess all these properties simultaneously. For example, property p1 is non satisfied for all the approaches discussed in Section 2.1 and 2.3, the approaches discusses in Section 2.2 either do no satisfy property p1 as in [Agostinelli et al, 2014] or property p3 as in [Scardapane et al, 2018].…”

Section: Summarizingmentioning

confidence: 99%

A simple and efficient architecture for trainable activation functions

Apicella¹,

Isgrò²,

Prevete³

2019

Neurocomputing

View full text Add to dashboard Cite

Learning automatically the best activation function for the task is an active topic in neural network research. At the moment, despite promising results, it is still difficult to determine a method for learning an activation function that is at the same time theoretically simple and easy to implement. Moreover, most of the methods proposed so far introduce new parameters or adopt different learning techniques. In this work we propose a simple method to obtain trained activation function which adds to the neural network local subnetworks with a small amount of neurons. Experiments show that this approach could lead to better result with respect to using a pre-defined activation function, without introducing a large amount of extra parameters that need to be learned.

show abstract

Kafnets: Kernel-based non-parametric activation functions for neural networks

Cited by 62 publications

References 37 publications

Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere

Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere

Efficient continual learning in neural networks with embedding regularization

A simple and efficient architecture for trainable activation functions

Contact Info

Product

Resources

About