The method of types [information theory]

Abstract

The method of types is one of the key technical tools in Shannon Theory, and this tool is valuable also in other fields. In this paper, some key applications will be presented in sufficient detail enabling an interested nonspecialist to gain a working knowledge of the method, and a wide selection of further applications will be surveyed. These range from hypothesis testing and large deviations theory through error exponents for discrete memoryless channels and capacity of arbitrarily varying channels to multiuser problems. While the method of types is suitable primarily for discrete memoryless models, its extensions to certain models with memory will also be discussed.Index Terms-Arbitrarily varying channels, choice of decoder, counting approach, error exponents, extended type concepts, hypothesis testing, large deviations, multiuser problems, universal coding.

A lower bound on the quantum capacity of channels with correlated errors

“…The classical (conditional) Kullback-Leibler information (informational divergence or relative entropy) is denoted by D and entropy by H. 10,11,22 Specifically, for a probability distribution Q on X 2 , transition (or conditional) probabilities P (v|u), u, v ∈ X , and a probability distribution p on X , we define…”

“…8, which employed the method of types 10,11,22,33 . In the present case, second-order (Markov) types rather than the usual types are used.…”

“…This method was exploited by the Hungarian mathematician (information theorist) Csiszár and coworkers around 1980 to present the strongest coding theorems such as the one showing the existence of universal channel codes asymptotically as good as any codes 10,11 . It has often produced results in elementary enumerative manners, which is also the case in this paper.…”

“…In the proof of the main result below, the method of types, which is a powerful tool from classical information theory, plays an important role 10,11 . This method was exploited by the Hungarian mathematician (information theorist) Csiszár and coworkers around 1980 to present the strongest coding theorems such as the one showing the existence of universal channel codes asymptotically as good as any codes 10,11 .…”

An exponential lower bound on the highest fidelity achievable by quantum error-correcting codes

“…(9) below] to deduce a lower bound on the quantum capacity. This work's approach resembles theirs in that both bounds are shown using random coding arguments [6,7,13] based on (9), but differs from [11] in that while [11] uses an analog of minimum Hamming distance decoding, this work employs an analog of minimum entropy decoding [14] together with the method of types from classical information theory [7,14,15], which enables us to obtain the exponential bound analogous to (1) in a simple enumerative manner.…”