Renyi's divergence and entropy rates for finite alphabet Markov sources

Abstract

In this work, we examine the existence and the computation of the Rényi divergence rate, lim (), between two time-invariant finite-alphabet Markov sources of arbitrary order and arbitrary initial distributions described by the probability distributions and , respectively. This yields a generalization of a result of Nemetz where he assumed that the initial probabilities under and are strictly positive. The main tools used to obtain the Rényi divergence rate are the theory of nonnegative matrices and Perron-Frobenius theory. We also provide numerical examples and investigate the limits of the Rényi divergence rate as 1 and as 0. Similarly, we provide a formula for the Rényi entropy rate lim () of Markov sources and examine its limits as 1 and as 0. Finally, we briefly provide an application to source coding.

Some properties of Rényi entropy and Rényi entropy rate

“…The Rényi entropy rate was first defined by Rached et al [25] for an ergodic Markov chain with a finite state space. The entropy rate of this process is expressed as…”

Uniform Random Number Generation From Markov Chains: Non-Asymptotic and Asymptotic Analyses

“…When the transition matrix has several irreducible components, we need to consider the mixture distribution among the possible irreducible components, which is defined by the initial distribution. As discussed in [54,Theorem 1], in the finite state space, the asymptotic behavior of the (conditional) Rényi entropy is characterized by the maximum (conditional) Rényi entropy among the possible irreducible components, which depend on the initial distribution. Hence, for large deviation and moderate deviation, the exponential decreasing rate of the leaked information can be evaluated by the minimum rate among the possible irreducible components.…”

The Rate of Entropy for Gaussian Processes

“…For example, the rate of Shannon entropy for a stationary Gaussian process was obtained by Kolmogorov (1958). The rate of Renyi entropy for stochastic processes was obtained by Rached et al (1999).…”

“…Up to now, the Renyi entropy rate for some stochastic processes have been studied, for example, by Rached et al (1999Rached et al ( , 2004 and Golshani and Pasha (2010). But in the case of the Tsallis entropy, the rate of entropy for stochastic processes has not been obtained for any processes yet.…”