Entropy, Information, and the Updating of Probabilities

Abstract

This paper is a review of a particular approach to the method of maximum entropy as a general framework for inference. The discussion emphasizes pragmatic elements in the derivation. An epistemic notion of information is defined in terms of its relation to the Bayesian beliefs of ideally rational agents. The method of updating from a prior to posterior probability distribution is designed through an eliminative induction process. The logarithmic relative entropy is singled out as a unique tool for updating (a) that is of universal applicability, (b) that recognizes the value of prior information, and (c) that recognizes the privileged role played by the notion of independence in science. The resulting framework—the ME method—can handle arbitrary priors and arbitrary constraints. It includes the MaxEnt and Bayes’ rules as special cases and, therefore, unifies entropic and Bayesian methods into a single general inference scheme. The ME method goes beyond the mere selection of a single posterior, and also addresses the question of how much less probable other distributions might be, which provides a direct bridge to the theories of fluctuations and large deviations.

Relating a System’s Hamiltonian to Its Entropy Production Using a Complex Time Approach

“…The Hamiltonian for a stationary black hole is conventionally given by its mass M BH within the Schwarzschild radius; yet, as is well known, any Hamiltonian can also be offset by a constant quantity (with there being no absolute value for the energy, see Caticha 2021 [ 20 ]), so that the value of the associated Hamiltonian as determined by Equation (27b) can be additionally offset by the energy lost to the Hawking radiation H 0 = M BH c 2 − H HR as required. Note that such an offset (by unity) is also seen in the Kramers–Kronig expression for the real part of the refractive index to obtain the correct Hilbert transform relationships (see, as an example, the unity offset in Equation (1.1) of Toll [ 16 ]).…”

Relating a System’s Hamiltonian to its Entropy Production Using a Complex-Time Approach

“…That is to say, the Hamiltonian (usually defined along the reversible t-axis) and the Entropy Production (defined in effect, empirically, along the irreversible -axis) are essentially two sides of the same coin, being Hilbert transform related (see Eqs. 10,11,14,15). What this means is that the choice of which axis to use to fully describe any physical phenomenon is somewhat arbitrary: depending only on the choice of metric.…”

“…The Hamiltonian for a stationary black hole is conventionally given by its mass MBH within the Schwarzschild radius; yet, as is well known, any Hamiltonian can also be offset by a constant quantity (there being no absolute value for the energy, see Caticha 2021 15 ) so that the value of the associated Hamiltonian as determined by Eq.23b can therefore be straightforwardly offset, in effect by H0=MBHc 2 as required. Note, such an offset is similarly seen in the Kramers-Kronig expression for the real part of the refractive index, which is always offset by unity in order to attain the correct Hilbert Transform relationships (see as an example the unity offset in Eq.1.1 of Toll [ref.5]).…”

Relating a System’s Hamiltonian to its Entropy Production Using a Complex-Time Approach

“…The Hamiltonian for a stationary black hole is conventionally given by its mass MBH within the Schwarzschild radius; yet, as is well known, any Hamiltonian can also be offset by a constant quantity (there being no absolute value for the energy, see Caticha 2021 19 ) so that the value of the associated Hamiltonian as determined by Eq.27b can therefore be offset by H0=MBHc 2 as required. Note, such an offset is also seen in the Kramers-Kronig expression for the real part of the refractive index, which is always offset by unity in order to attain the correct Hilbert Transform relationships (see as an example the unity offset in Eq.1.1 of Toll [ref.15]).…”