Simple random search provides a competitive approach to reinforcement learning

Abstract

A common belief in model-free reinforcement learning is that methods based on random search in the parameter space of policies exhibit significantly worse sample complexity than those that explore the space of actions. We dispel such beliefs by introducing a random search method for training static, linear policies for continuous control problems, matching state-ofthe-art sample efficiency on the benchmark MuJoCo locomotion tasks. Our method also finds a nearly optimal controller for a challenging instance of the Linear Quadratic Regulator, a classical problem in control theory, when the dynamics are not known. Computationally, our random search algorithm is at least 15 times more efficient than the fastest competing model-free methods on these benchmarks. We take advantage of this computational efficiency to evaluate the performance of our method over hundreds of random seeds and many different hyperparameter configurations for each benchmark task. Our simulations highlight a high variability in performance in these benchmark tasks, suggesting that commonly used estimations of sample efficiency do not adequately evaluate the performance of RL algorithms.

Parallelized and Randomized Adversarial Imitation Learning for Safety-Critical Self-Driving Vehicles

“…The ARS is a model-free reinforcement learning algorithm [39]. Based on randomized search in the parameter spaces of policies, the ARS executes the method of finite differences to control and adjust its weights and train the way how the policy performs its given tasks [47], [39]. Via the random search in the parameter spaces, the ARS algorithm conducts a derivativefree policy optimization with noises [47], [39].…”

“…Based on randomized search in the parameter spaces of policies, the ARS executes the method of finite differences to control and adjust its weights and train the way how the policy performs its given tasks [47], [39]. Via the random search in the parameter spaces, the ARS algorithm conducts a derivativefree policy optimization with noises [47], [39]. For updating the training weights in an effective way, the ARS (i) uniformly selects update directions and (ii) updates the policies based on the selected direction.…”

“…2N rollouts and rewards are collected by N noisy policies π t,k,± = θ t ± νδ k (line 10-12). The state normalization is used in RAIL (line 6, 19); and it makes policy π t,i,± have equal influence for the changes of state components when there are state components with various ranges [39], [38], [51]. Notice that the reason why state normalization is required is that the high dimensional problems have multiple state components with various ranges; and therefore it makes the policies to result in large changes in actions when the equivalent sized changes are not equally influence states.…”

Physics-informed Evolutionary Strategy based Control for Mitigating Delayed Voltage Recovery

“…Conventional deep RL methods rely on chain-rules and back-propagation to update the parameters of the neural networks, which makes it rather difficult to incorporate nondifferentiatable modules like TAM. Recently, evolutionary strategy (ES) has been proven to be an effective, scalable alternative to conventional RL methods [10], [11]. ES methods are derivative-free and can be easily parallelized.…”

“…where a t and cr t are the action and the learned criterion based on the current state s t and the current latent variable c t . The TAM is generated by comparing the voltage magnitude at each bus i with the voltage criterion cr t , as shown by (11). The action is filtered by conducting a element-wise multiplication with TAM, as shown by (12).…”