Differential Entropy Rate Characterisations of Long Range Dependent Processes

Abstract

A quantity of interest to characterise continuousvalued stochastic processes is the differential entropy rate. The rate of convergence of many properties of LRD processes is slower than might be expected, based on the intuition for conventional processes, e.g. Markov processes. Is this also true of the entropy rate?In this paper we consider the properties of the differential entropy rate of stochastic processes that have an autocorrelation function that decays as a power law. We show that power law decaying processes with similar autocorrelation and spectral density functions, Fractional Gaussian Noise and ARFIMA(0,d,0), have different entropic properties, particularly for negatively correlated parameterisations. Then we provide an equivalence between the mutual information between past and future and the differential excess entropy for stationary Gaussian processes, showing the finiteness of this quantity is the boundary between long and short range dependence. Finally, we analyse the convergence of the conditional entropy to the differential entropy rate and show that for short range dependence that the rate of convergence is of the order O(n −1 ), but it is slower for long range dependent processes and depends on the Hurst parameter.

Entropy Rate Bounds via Second-Order Statistics

“…In some specific Gaussian models, like the auto-regressive moving-average (ARMA) parametric family of processes, the integral in this formula boils down to a very simple expression [15]. Similar results have been lately proved for other important Gaussian models that exhibit long-range dependence [16]. Nonetheless, evaluation of exact values of entropy rates are a hard task in the general case.…”

“…The following result, which is proved in Appendix A, exhibits a generalization of the bound in (16) to discrete-valued stationary stochastic processes.…”

“…As can be seen in Figure 1, for relatively large λ values, the gap between the exact entropy and its upper bound is quite small. Since the upper bound (16) in the univariate case seems to be relatively tight, at least in some cases, it seems very natural to generalize it and propose an upper bound on the entropy rate of discrete-valued stationary stochastic processes, which merely depends on the second-order statistics of the process. Hence, the first objective of this work is to propose an upper bound on the entropy rates of discrete-valued stochastic processes in term of their PSD functions.…”