No abstract

No abstract

In this paper, approximation methods for binary polar code construction proposed by Tal and Vardy are extended to nonbinary source alphabets. Additionally, a new approximation method that enables accurate polar code construction with less usage of computational resources is proposed. Efficiency and accuracy of proposed methods are supported analytically and numerically.

We consider the reinforcement learning problem for partially observed Markov decision processes (POMDPs) with large or even countably infinite state spaces, where the controller has access to only noisy observations of the underlying controlled Markov chain. We consider a natural actor-critic method that employs a finite internal memory for policy parameterization, and a multi-step temporal difference learning algorithm for policy evaluation. We establish, to the best of our knowledge, the first non-asymptotic global convergence of actor-critic methods for partially observed systems under function approximation. In particular, in addition to the function approximation and statistical errors that also arise in MDPs, we explicitly characterize the error due to the use of finite-state controllers. This additional error is stated in terms of the total variation distance between the traditional belief state in POMDPs and the posterior distribution of the hidden state when using a finite-state controller. Further, we show that this error can be made small in the case of sliding-block controllers by using larger block sizes.

In a broad class of reinforcement learning applications, stochastic rewards have heavy-tailed distributions, which lead to infinite second-order moments for stochastic (semi)gradients in policy evaluation and direct policy optimization. In such instances, the existing RL methods may fail miserably due to frequent statistical outliers. In this work, we establish that temporal difference (TD) learning with a dynamic gradient clipping mechanism, and correspondingly operated natural actor-critic (NAC), can be provably robustified against heavy-tailed reward distributions. It is shown in the framework of linear function approximation that a favorable tradeoff between bias and variability of the stochastic gradients can be achieved with this dynamic gradient clipping mechanism. In particular, we prove that robust versions of TD learning achieve sample complexities of order O(εwith and without the full-rank assumption on the feature matrix, respectively, under heavy-tailed rewards with finite moments of order (1 + p) for some p ∈ (0, 1], both in expectation and with high probability. We show that a robust variant of NAC based on Robust TD learning achieves Õ(ε −4− 2 p ) sample complexity. We corroborate our theoretical results with numerical experiments.