Basis iteration for reward based dimensionality reduction

Sprague, Nathan

doi:10.1109/devlrn.2007.4354032

Cited by 5 publications

(4 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our case, observations can contain continuous features, which would require an infinite number of states. State vector transformation approaches project a high-dimensional space onto a low-dimensional space, such as basis iteration for reward based dimensionality reduction (Sprague, 2007 ), locally linear embedding (LLE) (Roweis and Saul, 2000 ) and dimensionality reduction by learning an invariant mapping (DrLIM) (Hadsell et al, 2006 ). Apart from DrLIM, these methods require recomputation of the embedding for each unknown datapoint, and rely on predetermined computable distance metrics.…”

Section: Related Workmentioning

confidence: 99%

“…A large state space increases computational costs and can lead to overfitting (Sỳkora, 2008 ). It is possible to reduce the state space through various methods, such as state clustering or segmentation [e.g., Q-learning with adaptive state segmentation (Murao and Kitamura, 1997 )], state vector transformation [e.g., basis iteration for reward based dimensionality reduction (Sprague, 2007 )] and state space reconstruction [e.g., action respecting embedding (Bowling et al, 2005 ; Sỳkora, 2008 )]. State space reduction can improve generalization, as well as reduce computational complexity and learning time.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Model Reduction Through Progressive Latent Space Pruning in Deep Active Inference

et al. 2022

View full text Add to dashboard Cite

Although still not fully understood, sleep is known to play an important role in learning and in pruning synaptic connections. From the active inference perspective, this can be cast as learning parameters of a generative model and Bayesian model reduction, respectively. In this article, we show how to reduce dimensionality of the latent space of such a generative model, and hence model complexity, in deep active inference during training through a similar process. While deep active inference uses deep neural networks for state space construction, an issue remains in that the dimensionality of the latent space must be specified beforehand. We investigate two methods that are able to prune the latent space of deep active inference models. The first approach functions similar to sleep and performs model reduction post hoc. The second approach is a novel method which is more similar to reflection, operates during training and displays “aha” moments when the model is able to reduce latent space dimensionality. We show for two well-known simulated environments that model performance is retained in the first approach and only diminishes slightly in the second approach. We also show that reconstructions from a real world example are indistinguishable before and after reduction. We conclude that the most important difference constitutes a trade-off between training time and model performance in terms of accuracy and the ability to generalize, via minimization of model complexity.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Model Reduction Through Progressive Latent Space Pruning in Deep Active Inference

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Dimension reduction is one method to deal with the freedom, and can be done via a developmental approach [94]. Other methods guide exploration by emotion or motivation based heuristics, such as curiosity or disappointment [95]- [98].…”

Section: Sensorimotor Controlmentioning

confidence: 99%

Biomimetic learning, not learning biomimetics: A survey of developmental learning

Jennings

Ordóñez

2010

Proceedings of the IEEE 2010 National Aerospace &Amp; Electronics Conference

View full text Add to dashboard Cite

Rather than try to learn skills directly, developmental learning seeks to create a general purpose system inspired by the human developmental process. The premise is that intelligent behavior can only be achieved through the course of experience, and attempts to circumvent this approach will lead to a plateau limiting performance. There are many facets of this approach and some of the common examples are categorization, sensorimotor control and social learning. These are relatively simple tasks for people, yet challenging for an autonomous robot. A survey of research in this recent field is presented. The concepts and the tools used are described so that other disciplines may have a understanding of this field and use the paradigm in their applications.

show abstract

“…However, it can lead to considerable information loss (Wang et al, 2017). Sprague (2007) proposed an iterative dimension reduction method using neighborhood components analysis. Their method uses a linear basis function to model the Q‐function and cannot allow more general nonlinear function approximation.…”

Section: Introductionmentioning

confidence: 99%

Deep spectral Q‐learning with application to mobile health

Gao

Shi

Song

2023

Stat

View full text Add to dashboard Cite

Dynamic treatment regimes assign personalized treatments to patients sequentially over time based on their baseline information and time‐varying covariates. In mobile health applications, these covariates are typically collected at different frequencies over a long time horizon. In this paper, we propose a deep spectral Q‐learning algorithm, which integrates principal component analysis (PCA) with deep Q‐learning to handle the mixed frequency data. In theory, we prove that the mean return under the estimated optimal policy converges to that under the optimal one and establish its rate of convergence. The usefulness of our proposal is further illustrated via simulations and an application to a diabetes dataset.

show abstract

Basis iteration for reward based dimensionality reduction

Cited by 5 publications

References 6 publications

Model Reduction Through Progressive Latent Space Pruning in Deep Active Inference

Model Reduction Through Progressive Latent Space Pruning in Deep Active Inference

Biomimetic learning, not learning biomimetics: A survey of developmental learning

Deep spectral Q‐learning with application to mobile health

Contact Info

Product

Resources

About