Learning Sparse Representations in Reinforcement Learning with Sparse Coding

Le, Lei; Kumaraswamy, Raksha; White, Martha

doi:10.24963/ijcai.2017/287

Cited by 15 publications

(6 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This involves penalising changes to weights deemed important for old tasks [14] or enforcing weight or representational sparsity [3] to ensure that only a subset of neurons remain active at any point of time. The latter method has been shown to reduce the possibility of catastrophic interference across tasks [15,26].…”

Section: Related Workmentioning

confidence: 99%

“…Besides aligning gradients, meta-learning algorithms show promise for CL since they can directly use the meta-objective to influence model optimisation and improve on auxiliary objectives like generalisation or transfer. This avoids having to define heuristic incentives like sparsity [15] for better CL. The downside is that they are usually slow and hard to tune, effectively rendering them more suitable for offline continual learning [12,22].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

La-MAML: Look-ahead Meta Learning for Continual Learning

Gupta,

Yadav,

Paull

2020

Preprint

View full text Add to dashboard Cite

The continual learning problem involves training models with limited capacity to perform well on a set of an unknown number of sequentially arriving tasks. While meta-learning shows great potential for reducing interference between old and new tasks, the current training procedures tend to be either slow or offline, and sensitive to many hyper-parameters. In this work, we propose Look-ahead MAML (La-MAML), a fast optimisation-based meta-learning algorithm for onlinecontinual learning, aided by a small episodic memory. Our proposed modulation of per-parameter learning rates in our meta-learning update allows us to draw connections to prior work on hypergradients and meta-descent. This provides a more flexible and efficient way to mitigate catastrophic forgetting compared to conventional prior-based methods. La-MAML achieves performance superior to other replay-based, prior-based and meta-learning based approaches for continual learning on real-world visual classification benchmarks. * equal contribution Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

La-MAML: Look-ahead Meta Learning for Continual Learning

Gupta,

Yadav,

Paull

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Also, it was determined that SR is the mechanism in the primary visual cortex to achieve concise description of images in terms of features and considered as a main principle to efficiently represent complex data [13], [14]. Furthermore, SR is used to enforce the learning process [15] and to solve optimization problems with non-convex objective function [16]. Hence, SR-based classification (SRC) [9]- [12] methods have proven to be efficient and robust to noise, occlusion, and corruption.…”

Section: Introductionmentioning

confidence: 99%

Image Classification Based on Sparse Representation in the Quaternion Wavelet Domain

et al. 2022

View full text Add to dashboard Cite

In this study, we propose a novel sparse representation learning method in the Quaternion Wavelet (QW) domain for multi-class image classification. The proposed method takes advantages from: i) the QW decomposition, which promotes sparsity and provides structural information about the image data while allowing approximate shift-invariance, to extract meaningful features from low-frequency QW subbands, ii) the dimensionality reduction method using Principal Component Analysis (PCA) for reducing the complexity of the problem, and iii) the sparse representation of the generated QW features to efficiently learn and capture the meaningful and compact information of this data. After the QW decomposition, the features extracted from low-frequency image sub-bands information are projected, by the PCA, into a new feature space with lower dimensionality. The features extracted from the training samples are used to construct a dictionary, while the features of the test samples are sparsely coded for the classification step. The sparse coding problem is formulated in a QW Least Absolute Shrinkage and Selection Operator (QWLasso) model applying quaternion l 1 minimization. A novel Quaternion Fast Iterative Shrinkage-Thresholding Algorithm (QFISTA) is developed to solve the QWLasso model. The experiments conducted on various public image datasets validated that the proposed method possesses higher accuracy, sparsity, and robustness in comparison with several contemporary methods in the field including Neural Networks.

show abstract

“…Early work comparing gradient TD algorithms [Maei et al, 2009] used sampled trajectories-2500 of them-but compared to returns, rather than value estimates. For several empirical studies using benchmark domains, like Mountain Car and Acrobot, there are a variety of choices, including t = m = 500 [Gehring et al, 2016]; m = 2000, t = 300 and 1000 length rollouts [Pan et al, 2017]; and m = 5000, t = 5000 [Le et al, 2017]. For a continuous physical system, [Dann et al, 2014] used as little as 10 rollouts from a state.…”

Section: Introductionmentioning

confidence: 99%

High-confidence error estimates for learned value functions

Sajed,

Chung,

White

2018

Preprint

Self Cite

View full text Add to dashboard Cite

Estimating the value function for a fixed policy is a fundamental problem in reinforcement learning. Policy evaluation algorithms-to estimate value functions-continue to be developed, to improve convergence rates, improve stability and handle variability, particularly for off-policy learning. To understand the properties of these algorithms, the experimenter needs high-confidence estimates of the accuracy of the learned value functions. For environments with small, finite state-spaces, like chains, the true value function can be easily computed, to compute accuracy. For large, or continuous state-spaces, however, this is no longer feasible. In this paper, we address the largely open problem of how to obtain these high-confidence estimates, for general statespaces. We provide a high-confidence bound on an empirical estimate of the value error to the true value error. We use this bound to design an offline sampling algorithm, which stores the required quantities to repeatedly compute value error estimates for any learned value function. We provide experiments investigating the number of samples required by this offline algorithm in simple benchmark reinforcement learning domains, and highlight that there are still many open questions to be solved for this important problem.

show abstract

Learning Sparse Representations in Reinforcement Learning with Sparse Coding

Cited by 15 publications

References 27 publications

La-MAML: Look-ahead Meta Learning for Continual Learning

La-MAML: Look-ahead Meta Learning for Continual Learning

Image Classification Based on Sparse Representation in the Quaternion Wavelet Domain

High-confidence error estimates for learned value functions

Contact Info

Product

Resources

About