Model-Based Relative Entropy Stochastic Search

Abstract

Stochastic search algorithms are general black-box optimizers. Due to their ease of use and their generality, they have recently also gained a lot of attention in operations research, machine learning and policy search. Yet, these algorithms require a lot of evaluations of the objective, scale poorly with the problem dimension, are affected by highly noisy objective functions and may converge prematurely. To alleviate these problems, we introduce a new surrogate-based stochastic search approach. We learn simple, quadratic surrogate models of the objective function. As the quality of such a quadratic approximation is limited, we do not greedily exploit the learned models. The algorithm can be misled by an inaccurate optimum introduced by the surrogate. Instead, we use information theoretic constraints to bound the 'distance' between the new and old data distribution while maximizing the objective function. Additionally the new method is able to sustain the exploration of the search distribution to avoid premature convergence. We compare our method with state of art black-box optimization methods on standard uni-modal and multi-modal optimization functions, on simulated planar robot tasks and a complex robot ball throwing task. The proposed method considerably outperforms the existing approaches.

Layered direct policy search for learning hierarchical skills

“…Both layers are connected by regression problems that are used to estimate the corresponding reward models. While we implemented learning algorithms that are based on the Model-Based Relative Entropy Stochastic Search (MORE) [14] algorithm for the individual layers, the whole framework is more general and other algorithms could be used that allow for the use of sample reweighting with importance weights.…”

“…The sub-policy optimization is an extension of ModelBased Relative Entropy Stochastic Search (MORE) [14]. Both algorithms consist of the two steps: first estimating a reward model and second updating the (sub-)policy.…”

Deriving and improving CMA-ES with information geometric trust regions

“…The use of trust regions is a common approach in policy search to obtain a stable policy update and to avoid premature convergence [1,17,18]. The model-based relative entropy stochastic search algorithm (MORE) [1] uses a local quadratic surrogate to update it's Gaussian search distribution.…”

“…The model-based relative entropy stochastic search algorithm (MORE) [1] uses a local quadratic surrogate to update it's Gaussian search distribution. As these surrogates can be inaccurate, MORE doesn't exploit the surrogate greedily, but uses a trust region in form of a KL-bound.…”

“…This search distribution is used to generate a population of individuals which are evaluated by their corresponding fitness values. Subsequently, a new search distribution is computed by either computing gradient based updates [19], expectation-maximisationbased updates [7,12], evolutionary strategies [10], the cross-entropy method [14] or information-theoretic policy updates [1], such that the individuals with higher fitness will have better selection probability. The Covariance Matrix Adaptation -Evolutionary Strategy (CMA-ES) is one of the most popular stochastic search algorithms [10].…”