A survey of Monte Carlo methods for noisy and costly densities with application to reinforcement learning

Abstract

This survey gives an overview of Monte Carlo methodologies using surrogate models, for dealing with densities which are intractable, costly, and/or noisy. This type of problem can be found in numerous real-world scenarios, including stochastic optimization and reinforcement learning, where each evaluation of a density function may incur some computationally-expensive or even physical (real-world activity) cost, likely to give different results each time. The surrogate model does not incur this cost, but there are important trade-offs and considerations involved in the choice and design of such methodologies. We classify the different methodologies into three main classes and describe specific instances of algorithms under a unified notation. A modular scheme which encompasses the considered methods is also presented. A range of application scenarios is discussed, with special attention to the likelihood-free setting and reinforcement learning. Several numerical comparisons are also provided.

Towards Automatic Facility Layout Design Using Reinforcement Learning

“…This is called the Markov property. There are three main learning methods for reinforcement learning: dynamic programming (DP) method [17], Monte Carlo (MC) method [18], and time difference (TD) method [19]. The DP method is a method of obtaining optimal behavior by solving the Bellman equation when each parameter of the system is known.…”

A Survey of Quantum Alternatives to Randomized Algorithms: Monte Carlo Integration and Beyond

“…Monte Carlo sampling (see, for example, [4]) is a set of techniques that randomly generate numerical quantities for the purpose of simulating a statistical distribution or computing a moment or other expectation thereof (e.g., mean, variance). It is prominent in many disciplines, including computational finance [5], computational physics [6], artificial intelligence [7,8], and various branches of engineering [9]. Although the concepts and ideas discussed in this paper readily generalize to other disciplines, we present our exposition with a focus on computational finance.…”

Optimality in Noisy Importance Sampling

“…A wide range of modern applications, especially in Bayesian inference framework [1], require the study of probability density functions (pdfs) which can be evaluated stochastically, i.e., only noisy evaluations can be obtained [2,3,4,5]. For instance, this is the case of the pseudo-marginal approaches and doubly intractable posteriors [6,7], approximate Bayesian computation (ABC) and likelihood-free schemes [8,9], where the target density cannot be computed in closed-form.…”