Fuzzy Baselines to Stabilize Policy Gradient Reinforcement Learning

Abstract

Policy gradient methods are amongst the most efficient for on-policy, model-free reinforcement learning. However, they suffer from high variance in gradient updates, making them unstable during training. Subtracting a baseline from the rewards is an effective strategy to reduce variance, such as in actorcritic models. This work presents a variation of the actor-critic model that uses a fuzzy system instead of a neural network to estimate the state value function. The fuzzy value approximation is inspired by previous value-based methods such as fuzzy Q-learning. Experiments with the cart-pole benchmark show that fuzzy value approximation outperforms several reinforcement learning algorithms in terms of sample-efficiency.