Low Complexity, Low Probability Patterns and Consequences for Algorithmic Probability Applications

Abstract

Developing new ways to estimate probabilities can be valuable for science, statistics, engineering, and other fields. By considering the information content of different output patterns, recent work invoking algorithmic information theory inspired arguments has shown that a priori probability predictions based on pattern complexities can be made in a broad class of input-output maps. These algorithmic probability predictions do not depend on a detailed knowledge of how output patterns were produced, or historical statistical data. Although quantitatively fairly accurate, a main weakness of these predictions is that they are given as an upper bound on the probability of a pattern, but many low complexity, low probability patterns occur, for which the upper bound has little predictive value. Here, we study this low complexity, low probability phenomenon by looking at example maps, namely a finite state transducer, natural time series data, RNA molecule structures, and polynomial curves. Some mechanisms causing low complexity, low probability behaviour are identified, and we argue this behaviour should be assumed as a default in the real-world algorithmic probability studies. Additionally, we examine some applications of algorithmic probability and discuss some implications of low complexity, low probability patterns for several research areas including simplicity in physics and biology, a priori probability predictions, Solomonoff induction and Occam’s razor, machine learning, and password guessing.

Exploring Simplicity Bias in 1D Dynamical Systems

“…An upper-bound fit to the log 10 P(x) data gives the slope as −0.18. Note that many output strings fall below the upper-bound prediction, as we expected based on earlier studies [1,37], but nonetheless it is known that randomly generated outputs tend to be close to the bound [2]. Put differently, even though many output strings (blue dots) appear to be far below the bound, most of the probability mass for each complexity value is concentrated close to the bound.…”

“…A weakness of our probability predictions is that they only constitute an upper bound on the probabilities, and, for example, Figure 3a shows that many output trajectory patterns x fall far below their respective upper bounds. Following the hypothesis from [2,37], these low-complexity, low-probability outputs are presumably patterns which the logistic map finds 'hard' to make, yet are not intrinsically very complex. Further, the presence of these low-complexity, low-probability patterns may indicate the non-universal power of the map [2].…”

“…Further, the presence of these low-complexity, low-probability patterns may indicate the non-universal power of the map [2]. A potential avenue for future work would be studying what types of patterns may occur far from the bound, and possible approaches to improving on their probability predictions [37].…”

“…The ways in which simplicity bias differs from Levin's coding theorem include that it does not assume UTMs, uses approximations of complexities, and for many outputs, P(x) ≪ 2 −K(x) . Hence, the abundance of low complexity, low-probability outputs [2,37] is a sign of simplicity bias.…”