Stochastic Decision Making in Learning Classifier Systems through a Natural Policy Gradient Method

Chen, Gang; Zhang, Mengjie; Pang, Shaoning; Douch, Colin I. J.

doi:10.1007/978-3-319-12643-2_37

Cited by 2 publications

(9 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Estimate θ t log π, µ t log π, and σ t log π (13) and (14), δ t can be computed straightforwardly from (24). Once θt logÃ [M ] s t is approximated by using (26) and (27), we can proceed to approximate µt logÃ [M ] s t and σt logÃ [M ] s t as well.…”

Section: A Policy Parameter Learning Componentmentioning

confidence: 99%

“…We have shown previously that it is often beneficial for an agent to learn stochastic policies or stochastic action selection strategies [21], [24]. For reinforcement learning in continuous spaces, a stochastic policy π( s t , a t ) at any time t describes a continuous probability distribution, from which the actual action a t to be performed by the agent will be sampled.…”

Section: Reinforcement Learning In Continuous Spacesmentioning

confidence: 99%

“…Calculate δt according to (24). Estimate θ t log π, µ t log π, and σ t log π (13) and (14), δ t can be computed straightforwardly from (24).…”

Section: A Policy Parameter Learning Componentmentioning

confidence: 99%

See 2 more Smart Citations

Accuracy-Based Learning Classifier Systems for Multistep Reinforcement Learning: A Fuzzy Logic Approach to Handling Continuous Inputs and Learning Continuous Actions

Chen

Douch

Zhang

2016

IEEE Trans. Evol. Computat.

Self Cite

View full text Add to dashboard Cite

Despite of the proven effectiveness, many Michigan learning classifier systems cannot perform multi-step reinforcement learning in continuous spaces. To meet this technical challenge, some learning classifier systems have been designed to learn fuzzy logic rules. They can be largely classified into strength-based and accuracy-based systems. The latter is gaining more research attention in the last decade. However existing accuracy-based learning systems either address primarily singlestep learning problems or require the action space to be discrete. In this paper, a new accuracy-based learning fuzzy classifier system is developed to explicitly handle continuous state input and continuous action output during multi-step reinforcement learning. Several technical improvements have been achieved while developing the new learning algorithm. Particularly, we have successfully extended Q-learning like credit assignment methods to continuous spaces. To enable direct learning of stochastic strategies for action selection, we have also proposed to use a new fuzzy logic system with stochastic action outputs. Moreover, fine-grained learning of fuzzy rules has been achieved effectively in our algorithm by using a natural gradient learning method. It is the first time for these techniques to be utilized substantially in any accuracy-based learning fuzzy classifier systems. Meanwhile, in comparison with several recently proposed learning algorithms, our algorithm is shown to perform highly competitively on four benchmark learning problems and a robotics problem. The practical usefulness of our algorithm is also demonstrated by improving the performance of a wireless body area network.

show abstract

Section: A Policy Parameter Learning Componentmentioning

confidence: 99%

Section: Reinforcement Learning In Continuous Spacesmentioning

confidence: 99%

See 1 more Smart Citation

Accuracy-Based Learning Classifier Systems for Multistep Reinforcement Learning: A Fuzzy Logic Approach to Handling Continuous Inputs and Learning Continuous Actions

Chen

Douch

Zhang

2016

IEEE Trans. Evol. Computat.

Self Cite

View full text Add to dashboard Cite

show abstract

“…In general, however, G is a function of ω. In the Euclidean space, learning of ω is carried out through the updating rule in (19). In the Riemannian space, on the other hand, according to [1], learning should be performed based on the natural gradient of J, i.e.˜ ωt J.…”

Section: Learnpolicyparameters() {mentioning

confidence: 99%

“…Instead of learning deterministic policies, an agent that learns stochastic policies may easily escape from a local loop and hence potentially present a good solution to the above problem [19]. This idea can be realized by using, for example, a probabilistic action selection strategy where the chance of performing any action in a state is made proportional to its expected payoff (i.e.…”

Section: Introductionmentioning

confidence: 99%

Using Learning Classifier Systems to Learn Stochastic Decision Policies

Chen

Douch

Zhang

2015

IEEE Trans. Evol. Computat.

Self Cite

View full text Add to dashboard Cite

To solve reinforcement learning problems, many learning classifier systems are designed to learn state-action value functions through a compact set of maximally general and accurate rules. Most of these systems focus primarily on learning deterministic policies by using a greedy action selection strategy. However, in practice, it may be more flexible and desirable to learn stochastic policies, which can be considered as direct extensions of their deterministic counterparts. In this paper, we aim to achieve this goal by extending each rule with a new policy parameter. Meanwhile, a new method for adaptive learning of stochastic action selection strategies based on a policy gradient framework has also been introduced. Using this method, we have developed two new learning systems, one based on a regular gradient learning technology and the other based on a new natural gradient learning method. Both learning systems have been evaluated on three different types of reinforcement learning problems. The promising performance of the two systems clearly shows that learning classifier systems provide a suitable platform for efficient and reliable learning of stochastic policies.

show abstract

Stochastic Decision Making in Learning Classifier Systems through a Natural Policy Gradient Method

Cited by 2 publications

References 8 publications

Accuracy-Based Learning Classifier Systems for Multistep Reinforcement Learning: A Fuzzy Logic Approach to Handling Continuous Inputs and Learning Continuous Actions

Accuracy-Based Learning Classifier Systems for Multistep Reinforcement Learning: A Fuzzy Logic Approach to Handling Continuous Inputs and Learning Continuous Actions

Using Learning Classifier Systems to Learn Stochastic Decision Policies

Contact Info

Product

Resources

About