Regularized covariance estimation for weighted maximum likelihood policy search methods

Abstract

Abstract-Many episode-based (or direct) policy search algorithms, maintain a multivariate Gaussian distribution as search distribution over the parameter space of some objective function. One class of algorithms, such as episodic REPS, PoWER or PI 2 uses, a weighted maximum likelihood estimate (WMLE) to update the mean and covariance matrix of this distribution in each iteration. However, due to high dimensionality of covariance matrices and limited number of samples, the WMLE is an unreliable estimator. The use of WMLE leads to overfitted covariance estimates, and, hence the variance/entropy of the search distribution decreases too quickly, which may cause premature convergence. In order to alleviate this problem, the estimated covariance matrix can be regularized in different ways, for example by using a convex combination of the diagonal covariance estimate and the sample covariance estimate. In this paper, we propose a new covariance matrix regularization technique for policy search methods that uses the convex combination of the sample covariance matrix and the old covariance matrix used in last iteration. The combination weighting is determined by specifying the desired entropy of the new search distribution. With this mechanism, the entropy of the search distribution can be gradually decreased without damage from the maximum likelihood estimate.

Deriving and improving CMA-ES with information geometric trust regions

“…While this algorithm can improve the search distribution, it can quickly result in a degenerated distribution which stops exploration and would lead to premature convergence, which is a problem in many of these methods [12,14]. The cause of this limitation is mainly the maximum likelihood estimate (Equation 10) which overfits the current individuals and change the current search distribution drastically [2].…”

“…2 We 2 In this paper σ is used to denote the variance which we refer to it as step size. seek to find a distribution over individuals x, denoted π (x; θ ), that minimises the expected fitness…”

“…For completing the hole reaching task, the robot's end effector has to reach the bottom of a hole (35cm wide and 1m deep) centered point [2,0] without any collision with the ground or the hole wall. The reward was given by a quadratic cost term for the distance to bottom of the hole, quadratic costs for high accelerations and quadratic costs for collisions with the environment.…”

“…The maximisation of lower bound in each iteration is equivalent to weighted maximum likelihood estimation (MLE) of the distribution. It is well-known that ML estimation of the covariance matrix yields a degenerate, over-fitted distribution [2] which typically results in premature convergence of the algorithm. We extend the EM-based framework by incorporating information geometric trust regions.…”

Non-parametric contextual stochastic search

“…In addition, REPS and RBF-REPS can suffer from premature convergence. In [12] Covariance Estimation with Controlled Entropy Reduction (CECER) algorithm was introduced to alleviate the premature convergence problem of REPS. Our new algorithm local CECER leverage from both nonlinear generalization over contexts and fully context dependent search distribution update rule while it also uses CECER algorithm concept [12] to avoid premature convergence.…”