Directed Data-Processing Inequalities for Systems with Feedback

Abstract

We present novel data-processing inequalities relating the mutual information and the 1 directed information in systems with feedback. The internal blocks within such systems are restricted 2 only to be causal mappings, but are allowed to be non-linear, stochastic and time varying. These blocks 3 can for example represent source encoders, decoders or even communication channels. Moreover, 4 the involved signals can be arbitrarily distributed. Our first main result relates mutual and directed 5 informations and can be interpreted as a law of conservation of information flow. Our second main 6 result is a pair of data-processing inequalities (one the conditional version of the other) between 7 nested pairs of random sequences entirely within the closed loop. Our third main result is introducing 8 and characterizing the notion of in-the-loop (ITL) transmission rate for channel coding scenarios 9 in which the messages are internal to the loop. Interestingly, in this case the conventional notions 10 of transmission rate associated with the entropy of the messages and of channel capacity based on 11maximizing the mutual information between the messages and the output turn out to be inadequate.12 Instead, as we show, the ITL transmission rate is the unique notion of rate for which a channel code 13 attains zero error probability if and only if such ITL rate does not exceed the corresponding directed 14 information rate from messages to decoded messages. We apply our data-processing inequalities to 15 show that the supremum of achievable (in the usual channel coding sense) ITL transmission rates 16 is upper bounded by the supremum of the directed information rate across the communication 17 channel. Moreover, we present an example in which this upper bound is attained. Finally, we further 18 illustrate the applicability of our results by discussing how they make possible the generalization of 19 two fundamental inequalities known in networked control literature.