An investigation of methods of parameter tuning for Q-Learning Fuzzy Inference System

Abstract

This paper investigates four methods of implementing a Q-Learning Fuzzy Inference System(QFIS) algorithm to autonomously tune the parameters of a fuzzy inference system. We use an actor-critique structure and we simulate mobile robots playing the differential form of the pursuit evasion game. Both the critique and the actor are fuzzy inference systems. The four methods come from the fact whether it is necessary to tune all the parameters (i.e. all the premise and the consequent parameters) of the critique and the actor or just tune their consequent parameters. The four methods are applied to three versions of the pursuit evasion games. In the first version just the pursuer is learning. In the second version, the evader uses its higher maneuverability and plays intelligently against a self-learning pursuer. In the final version, both the pursuer and the evader are learning. We evaluate which parameters are best to tune and which parameters have little impact on the performance.

Learning in the Multi-Robot Pursuit Evasion Game

“…Each input has three Gaussian membership functions (MFs), with linguistic values of P (positive), Z (zero) and N (negative). Thus, depending on the number of inputs and their corresponding MFs, the FLC has 21 parameters that can be tuned during the learning phase; the tunable parameters are explained in the previous work [137]. The fuzzy output u i is defuzzified into crisp output using the weighted average defuzzification method [138].…”

“…For each learning player, the FIS is used to approximate the action-value function, whereas the FLC is used to find its control signal. In [137] four methods of QLFIS parameter tuning were investigated in order to reduce the computational time without effecting the overall performance of the learning algorithm. In the first method, only the consequent parameters of the FLC and the FIS were tuned, while in the second method the consequent parameters of the FLC and all the parameters (i.e., the premise and the consequent parameters) of the FIS were tuned.…”

“…In this game, it is assumed that the evader plays its DCS as defined in [137]. It is also assumed that the pursuer does not have any information about the evader's control strategy.…”

“…For the learning agent i, the FIS consists of two inputs and one output. The two inputs are z 1 and z 2 that correspond to the angle difference δ i and its derivativeδ i , respectively [137]. The output for the actor is the steering angle u i , while the output of the critic is the estimate of V i (s), where δ i is given by…”

“…where (x e , y e ) and (x p , y p ) are the positions of the evader and pursuer, and θ i is the orientation of learning agent i. be tuned during the learning phase. The tuned parameters are the means and the standard deviations of all the input MFs and the consequent parameters of the fuzzy rule base, as given in [137]. The fuzzy output is defuzzified into a crisp output using the weighted average defuzzification method [138].…”