An Operational Characterization of Mutual Information in Algorithmic Information Theory

Abstract

We show that the mutual information, in the sense of Kolmogorov complexity, of any pair of strings x and y is equal, up to logarithmic precision, to the length of the longest shared secret key that two parties, one having x and the complexity profile of the pair and the other one having y and the complexity profile of the pair, can establish via a probabilistic protocol with interaction on a public channel. For ℓ > 2, the longest shared secret that can be established from a tuple of strings (x1, ..., x ℓ ) by ℓ parties, each one having one component of the tuple and the complexity profile of the tuple, is equal, up to logarithmic precision, to the complexity of the tuple minus the minimum communication necessary for distributing the tuple to all parties. We establish the communication complexity of secret key agreement protocols that produce a secret key of maximal length, for protocols with public randomness. We also show that if the communication complexity drops below the established threshold, then only very short secret keys can be obtained. supported in part by the National Science Foundation through grant 1811729. No protocol can produce a longer shared secret key (up to an O(log n) additive term).Secret key agreement for three or more parties. Mutual information is only defined for two strings, but secret key agreement can be explored for the case of more strings. Let us consider again an example. Suppose that each of Alice, Bob, and Charlie have a point in the affine plane over the finite field with 2 n elements, and that the three points, which we call A, B, C, are collinear. Thus, each party has 2n bits of information, but together they have 5n bits of information, because given two points, the third one can be described with n bits. The parties want to establish a common secret key, but they can only communicate by broadcasting messages over a public channel. They can proceed as follows. Alice will broadcast a string p A , Bob a string p B , and Charlie a string p C , such that each party using his/her point and the received information will reconstruct the three collinear points A, B, C. A protocol that achieves this is called an omniscience protocol because it spreads to everyone the information possessed at the beginning individually by each party. In the next step, each party will compress the 5n bits, comprising the three points, to a string that is random given p A , p B , p C . The compressed string is the common secret key. We will see that up to logarithmic precision it has length 5n − (|p A | + |p B | + |p C |). Assuming we know how to do the omniscience protocol and the compression step, this protocol produces a common secret key of length 5n − CO(A, B, C), where CO (A, B, C) is the minimum communication for the omniscience task for the points A, B, C. In our example, it is clear that each one of p A , p B , p C must be at least n bits long, and that any two of these strings must contain together at least 3n bits. Using some recent results from the reference [Zim17], it can be shown that an...

Secret key agreement from correlated data, with no prior information

“…In case we use Kolmogorov complexity to measure the amount of information, I(x : y) is defined as C(x)+C(y)−C(x, y), and, up to logarithmic precision, is also equal to C(x)−C(x | y) and to C(y) − C(y | x). It is shown in [RZ18] (extending a classical result from [AC93] which is valid for inputs generated by memoryless processes, and which is using Shannon entropy to measure information), that no computable protocol (even probabilistic) can obtain a shared secret key longer than the mutual information of the inputs x and y. On the other hand, a protocol is presented in [RZ18] that with high probability produces a shared secret of length I(x : y) (up to logarithmic precison), provided, as mentioned above, the two parties know the complexity profile of the inputs.…”

“…It is shown in [RZ18] (extending a classical result from [AC93] which is valid for inputs generated by memoryless processes, and which is using Shannon entropy to measure information), that no computable protocol (even probabilistic) can obtain a shared secret key longer than the mutual information of the inputs x and y. On the other hand, a protocol is presented in [RZ18] that with high probability produces a shared secret of length I(x : y) (up to logarithmic precison), provided, as mentioned above, the two parties know the complexity profile of the inputs. Thus, the above discussion suggests that it is natural to aim for a shared secret key whose length is equal to the mutual information of the inputs, for some concept of information that measures the detectable correlation.…”

“…Leung [LYC76], Bennett et al [BBR88], Maurer [Mau93], Ahlswede and Csiszár [AC93] have started an extensive research line dedicated to the case when x and y are generated by a stochastic process, whose properties describe their correlation (see the survey of Narayan and Tyagi [NT16]). Recently, Romashchenko and Zimand [RZ18] have studied this problem in the very general framework of Algorithmic Information Theory using Kolmogorov complexity to gauge correlation without using any generative model for the provenance of x and y. 1 In all these works, Alice and Bob possess at the beginning of the protocol, in addition to x and y, some information about how these strings are correlated.…”

“…For instance, in the above example, Bob knows that the point P is on the line L. In the scenarios based on generative models, Alice and Bob know various attributes of the joint distribution of the two random variables (X, Y ) which describe the stochastic process that generates the pair (x, y), such as entropy, ergodic properties, etc. In the algorithmic information theory setting used in [RZ18], Alice and Bob know the complexity profile of (x, y), which is the 3-tuple (C(x), C(y), C(x, y)), where C(•) denotes Kolmogorov complexity. (Throughout this paper, C(x), called the Kolmogorov complexity of x or the minimal description length of x, is the length of a shortest program that when executed by a universal Turing machine prints x.…”

“…which is very large. The protocol in [RZ18] (in which the parties also hold the complexity profile of (x, y)) has communication complexity roughly C(x | y), which is shown to be optimal. Thus, in our case, it would be desirable to have a protocol with communication complexity close to C S(n) (x | y).…”