Interactive Learning in Continuous Multimodal Space: A Bayesian Approach to Action-Based Soft Partitioning and Learning

Firouzi, Hadi; Ahmadabadi, Majid Nili; Araabi, Babak Nadjar; Amizadeh, Saeed; Mirian, Maryam S.; Siegwart, Roland

doi:10.1109/tamd.2011.2170213

Cited by 10 publications

(16 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This problem is known as the curse of dimensionality. One way to tackle this problem is to use the experiences in the subspaces of ”, such as O i , for decision making [11] , [12] . However, the environment in the eyes of O i is partially observable, which creates a many-to-one mapping between real states of the environment and observations in O i .…”

Section: Methodsmentioning

confidence: 99%

Reward Maximization Justifies the Transition from Sensory Selection at Childhood to Sensory Integration at Adulthood

2014

Self Cite

View full text Add to dashboard Cite

In a multisensory task, human adults integrate information from different sensory modalities -behaviorally in an optimal Bayesian fashion- while children mostly rely on a single sensor modality for decision making. The reason behind this change of behavior over age and the process behind learning the required statistics for optimal integration are still unclear and have not been justified by the conventional Bayesian modeling. We propose an interactive multisensory learning framework without making any prior assumptions about the sensory models. In this framework, learning in every modality and in their joint space is done in parallel using a single-step reinforcement learning method. A simple statistical test on confidence intervals on the mean of reward distributions is used to select the most informative source of information among the individual modalities and the joint space. Analyses of the method and the simulation results on a multimodal localization task show that the learning system autonomously starts with sensory selection and gradually switches to sensory integration. This is because, relying more on modalities -i.e. selection- at early learning steps (childhood) is more rewarding than favoring decisions learned in the joint space since, smaller state-space in modalities results in faster learning in every individual modality. In contrast, after gaining sufficient experiences (adulthood), the quality of learning in the joint space matures while learning in modalities suffers from insufficient accuracy due to perceptual aliasing. It results in tighter confidence interval for the joint space and consequently causes a smooth shift from selection to integration. It suggests that sensory selection and integration are emergent behavior and both are outputs of a single reward maximization process; i.e. the transition is not a preprogrammed phenomenon.

show abstract

Section: Methodsmentioning

confidence: 99%

Reward Maximization Justifies the Transition from Sensory Selection at Childhood to Sensory Integration at Adulthood

2014

Self Cite

View full text Add to dashboard Cite

show abstract

“…Using different measures of expertness for agents with different experiences, the experiment results show that certain expertness measures are less sensitive when the knowledge is incorrect and data is uncertain. The uncertainty over the states can also be addressed using an automatic action-based soft partitioning of the state space [5]. This approach uses multiple agents to suggest actions based on partial observations of the state space.…”

Section: Related Workmentioning

confidence: 99%

“…RL problems with state uncertainties are addressed by [4], [5], [6], [3]. [2] addresses uncertainties over credit assignment.…”

Section: Introductionmentioning

confidence: 99%

Fast Reinforcement Learning under Uncertainties with Self-Organizing Neural Networks

Teng

Tan

2015

2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT)

View full text Add to dashboard Cite

Using feedback signals from the environment, a reinforcement learning (RL) system typically discovers action policies that recommend actions effective to the states based on a Q-value function. However, uncertainties over the estimation of the Q-values can delay the convergence of RL. For fast RL convergence by accounting for such uncertainties, this paper proposes several enhancements to the estimation and learning of the Q-value using a self-organizing neural network. Specifically, a temporal difference method known as Q-learning is complemented by a Q-value Polarization procedure, which contrasts the Q-values using feedback signals on the effect of the recommended actions. The polarized Q-values are then learned by the self-organizing neural network using a Bi-directional Template Learning procedure. Furthermore, the polarized Qvalues are in turn used to adapt the reward vigilance of the ART-based self-organizing neural network using a Bi-directional Adaptation procedure. The efficacy of the resultant system called Fast Learning (FL) FALCON is illustrated using two singletask problem domains with large MDPs. The experiment results from these problem domains unanimously show FL-FALCON converging faster than the compared approaches.

show abstract

“…In the seventh category, our group exploited possible generalization in the subspaces to have faster learning (see [13], [14]). Here a subspace refers to a sub-dimension of the original state representation, which is called the full-space.…”

Section: Introductionmentioning

confidence: 99%

“…• We explain the idea of using subspaces in RL framework for expediting the learning process in a concrete model. In the previous works by other members of our group, the idea of using subspaces for expediting the learning process has been exploited [13], [14]. However, the idea was not exploited in a mathematically rigorous manner.…”

Section: Introductionmentioning

confidence: 99%

Exploiting Generalization in the Subspaces for Faster Model-Based Reinforcement Learning

Hashemzadeh

Hosseini

Ahmadabadi

2019

IEEE Trans. Neural Netw. Learning Syst.

Self Cite

View full text Add to dashboard Cite

Due to the lack of enough generalization in the statespace, common methods in Reinforcement Learning (RL) suffer from slow learning speed especially in the early learning trials. This paper introduces a model-based method in discrete statespaces for increasing the learning speed in terms of required experience (but not required computational time) by exploiting generalization in the experiences of the subspaces. A subspace is formed by choosing a subset of features in the original state representation (full-space). Generalization and faster learning in a subspace are due to many-to-one mapping of experiences from the full-space to each state in the subspace. Nevertheless, due to inherent perceptual aliasing in the subspaces, the policy suggested by each subspace does not generally converge to the optimal policy. Our approach, called Model Based Learning with Subspaces (MoBLeS), calculates confidence intervals of the estimated Q-values in the full-space and in the subspaces. These confidence intervals are used in the decision making, such that the agent benefits the most from the possible generalization while avoiding from detriment of the perceptual aliasing in the subspaces. Convergence of MoBLeS to the optimal policy is theoretically investigated. Additionally, we show through several experiments that MoBLeS improves the learning speed in the early trials.Index Terms-reinforcement learning, generalization in subspaces, learning speed, curse of dimensionality, value interval estimation arXiv:1710.08012v2 [stat.ML]

show abstract

Interactive Learning in Continuous Multimodal Space: A Bayesian Approach to Action-Based Soft Partitioning and Learning

Cited by 10 publications

References 25 publications

Reward Maximization Justifies the Transition from Sensory Selection at Childhood to Sensory Integration at Adulthood

Reward Maximization Justifies the Transition from Sensory Selection at Childhood to Sensory Integration at Adulthood

Fast Reinforcement Learning under Uncertainties with Self-Organizing Neural Networks

Exploiting Generalization in the Subspaces for Faster Model-Based Reinforcement Learning

Contact Info

Product

Resources

About