No abstract

Information-theoretic methodologies are increasingly being used in various disciplines. Frequently an information measure is adapted for a problem, yet the perspective of information as the unifying notion is overlooked. We set forth this perspective through presenting information-theoretic methodologies for a set of problems in probability and statistics. Our focal measures are Shannon entropy and Kullback-Leibler information. The background topics for these measures include notions of uncertainty and information, their axiomatic foundation, interpretations, properties, and generalizations. Topics with broad methodological applications include discrepancy between distributions, derivation of probability models, dependence between variables, and Bayesian analysis. More specific methodological topics include model selection, limiting distributions, optimal prior distribution and design of experiment, modeling duration variables, order statistics, data disclosure, and relative importance of predictors. Illustrations range from very basic to highly technical ones that draw attention to subtle points.

No abstract

This paper shows that multivariate distributions can be characterized as maximum entropy (ME) models based on the well-known general representation of density function of the ME distribution subject to moment constraints. In this approach, the problem of ME characterization simplifies to the problem of representing the multivariate density in the ME form, hence there is no need for case-by-case proofs by calculus of variations or other methods. The main vehicle for this ME characterization approach is the information distinguishability relationship, which extends to the multivariate case. Results are also formulated that encapsulate implications of the multiplication rule of probability and the entropy transformation formula for ME characterization. The dependence structure of multivariate ME distribution in terms of the moments and the support of distribution is studied. The relationships of ME distributions with the exponential family and with bivariate distributions having exponential family conditionals are explored. Applications include new ME characterizations of many bivariate distributions, including some singular distributions.