Directed Information, Causal Estimation, and Communication in Continuous Time

Abstract

A notion of directed information between two continuous-time processes is proposed. A key component in the definition is taking an infimum over all possible partitions of the time interval, which plays a role no less significant than the supremum over "space" partitions inherent in the definition of mutual information. Properties and operational interpretations in estimation and communication are then established for the proposed notion of directed information. For the continuous-time additive white Gaussian noise channel, it is shown that Duncan's classical relationship between causal estimation error and mutual information continues to hold in the presence of feedback upon replacing mutual information by directed information. A parallel result is established for the Poisson channel. The utility of this relationship is demonstrated in computing the directed information rate between the input and output processes of a continuous-time Poisson channel with feedback, where the channel input process is constrained to be constant between events at the channel output. Finally, the capacity of a wide class of continuous-time channels with feedback is established via directed information, characterizing the fundamental limit on reliable communication.

On the mutual information of time reversal for non-stationary channels

“…Interestingly, for our Gaussian channel model, the DI and MI are equal, which at first is somewhat surprising (as often in channels where feedback is employed, DI is strictly less than MI). While this may be able to be shown by relating / extending the proof of [18] [Proposition 3, part 3)] from continuous time to discrete time, and from scalar to vector observations, we prove it directly using linear algebra. …”

Minimax filtering regret via relations between information and estimation

“…We show that they are still valid in the presence of feedback by substituting mutual information with the notion of directed information in some cases as in continuous time developed in [6].…”

A View of Information-Estimation Relations in Gaussian Networks

“…where the last inequality follows by upper bounding the integral over [0, snr 0 ] by the LMMSE bound in (11) and by upper bounding the integral over [snr 0 , snr] using the SCPP bound in (21). Figure 7 shows a plot of C ∞ (snr, snr 0 , β) in (54) normalized by the capacity of the point-to-point channel 1 2 log(1 + snr).…”

“…The I-MMSE relationship has been extended to continuous time channels in [8] and generalized in [18] by using Malliavin calculus. For other continuous time generalizations the reader is referred to [19][20][21]. Finally, Venkat and Weissman [22] dispensed with the expectation and provided a point-wise identity that has given additional insight into this relationship.…”