An Online Kernel-Based Clustering Approach for Value Function Approximation

Abstract

Abstract. Value function approximation is a critical task in solving Markov decision processes and accurately representing reinforcement learning agents. A significant issue is how to construct efficient feature spaces from agent's samples in order to obtain optimal policy. This study addresses this challenge by proposing an online kernel-based clustering approach for building appropriate basis functions during the learning process. The method uses a kernel function capable of handling pairs of state-action as sequentially generated by the agent. At each time step, the procedure either adds new clusters, or adjusts the winning cluster's parameters. By considering the value function as a linear combination of the constructed basis functions, the weights are simultaneously optimized in a temporal-difference framework in order to minimize the Bellman approximation error. The proposed method is evaluated in numerous known simulated environments.