Existence and Continuity of Differential Entropy for a Class of Distributions

The sparsity and compressibility of finite-dimensional signals are of great interest in fields such as compressed sensing. The notion of compressibility is also extended to infinite sequences of i.i.d. or ergodic random variables based on the observed error in their nonlinear k-term approximation. In this work, we use the entropy measure to study the compressibility of continuous-domain innovation processes (alternatively known as white noise). Specifically, we define such a measure as the entropy limit of the doubly quantized (time and amplitude) process. This provides a tool to compare the compressibility of various innovation processes. It also allows us to identify an analogue of the concept of "entropy dimension" which was originally defined by Rényi for random variables. Particular attention is given to stable and impulsive Poisson innovation processes. Here, our results recognize Poisson innovations as the more compressible ones with an entropy measure far below that of stable innovations. While this result departs from the previous knowledge regarding the compressibility of fat-tailed distributions, our entropy measure ranks stable innovations according to their tail decay. Index TermsCompressibility, entropy, impulsive Poisson process, stable innovation, white Lévy noise. Symbol DefinitionSets Caligraphic letters like A, C, D, . . . Real and natural numbers R, NBorel sets in R B(R) or just B Random variablesCapital letters like: A, X, Y, Z, . . . Probability density function (pdf) of (continous) X p X or q X (lower-case letter p)Probability mass function (pmf) of (discrete) X P X (upper-case letter P )Cumulative distribution function (cdf) of X F X X 0 for a given white noise X(t) 1 0 X(t) dt; a random variable Definition 2 (Discrete Random Variable). [23] A random variable X is called discrete if it takes values in a countable alphabet set X ⊂ R. Definition 3 (Discrete-Continuous Random Variable).[23] A random variable X is called discrete-continuous with parameters (p c , P D , Pr {X ∈ D}) if there exists a countable set D, a discrete probability mass function P D , whose support is D, and a pdf p c ∈ AC such that 0 < Pr {X ∈ D} < 1,

show abstract

“…Definition 5. [25] Given α, ℓ, v ∈ (0, ∞), we define (α, ℓ, v)-AC to be the class of all p ∈ AC such that the corresponding density function p :…”

Section: B Definition Of Entropymentioning

confidence: 99%

How Compressible Are Innovation Processes?

Ghourchian

Amini

Gohari

2018

IEEE Trans. Inform. Theory

Self Cite

View full text Add to dashboard Cite

show abstract

“…The reason is that from (22) and (23), we see that {f (k) Y } and fŶ are uniformly bounded and have finite γ-moments. Therefore, (31) follows from [19,Theorem 1]. Thus, in step 2, we obtain that the sequence h (Y k ) has a limit.…”

Section: As a Result µ (K)mentioning

confidence: 92%

“…In this case, we claim that the support of the optimal solution only needs to have two members. To this end, note that the following problem is equivalent to the original problem defined in (19):…”

Section: Proof Of Lemmamentioning

confidence: 99%

See 1 more Smart Citation

On the Capacity of a Class of Signal-Dependent Noise Channels

Ghourchian

Aminian

Gohari

et al. 2018

IEEE Trans. Inform. Theory

Self Cite

View full text Add to dashboard Cite

In some applications, the variance of additive measurement noise depends on the signal that we aim to measure. For instance, additive Gaussian signal-dependent noise (AGSDN) channel models are used in molecular and optical communication. Herein we provide lower and upper bounds on the capacity of additive signal-dependent noise (ASDN) channels. The idea of the first lower bound is the extension of the majorization inequality, and for the second one, it uses some calculations based on the fact that h (Y ) > h (Y |Z). Both of them are valid for all additive signal-dependent noise (ASDN) channels defined in the paper. The upper bound is based on a previous idea of the authors ("symmetric relative entropy") and is used for the additive Gaussian signal-dependent noise (AGSDN) channels. These bounds indicate that in ASDN channels (unlike the classical AWGN channels), the capacity does not necessarily become larger by making the variance function of the noise smaller. We also provide sufficient conditions under which the capacity becomes infinity. This is complemented by a number of conditions that imply capacity is finite and a unique capacity achieving measure exists (in the sense of the output measure).

show abstract

“…For an infinite support domain, it must be considered that L → ∞. Discussions about existence and convergence of S d are presented in [19].…”

Section: Discrete and Continuous Complexity Measuresmentioning

confidence: 99%

Complexity measures for probability distributions with infinite domains

Rizzi

Piqueira

2021

Eur. Phys. J. B

View full text Add to dashboard Cite

Since the second half of the last century, the concept of complexity has been studied to find and connect ideas from different disciplines. Several quantifying methods have been proposed, based on computational measures extended to the context of biological and human sciences, as, for instance, the López-Ruiz, Mancini, and Calbet (LMC); and Shiner, Davison, and Landsberg (SDL) complexity measures, which take the concept of information entropy as the core of the definitions. However, these definitions are restricted to discrete probability distributions with finite domains, limiting the systems to be studied. Extensions of these measures were proposed for continuous probability distributions, but discrete distributions with infinite domains were not discussed. Here, these cases are studied and several distributions are analyzed, including the Zipf distribution, considered the paradigmatic model for self-organizing criticality.

show abstract

Existence and Continuity of Differential Entropy for a Class of Distributions

Cited by 23 publications

References 15 publications

How Compressible Are Innovation Processes?

How Compressible Are Innovation Processes?

On the Capacity of a Class of Signal-Dependent Noise Channels

Complexity measures for probability distributions with infinite domains

Contact Info

Product

Resources

About