Bounded Rational Decision-Making with Adaptive Neural Network Priors

Abstract

Bounded rationality investigates utility-optimizing decisionmakers with limited information-processing power. In particular, information theoretic bounded rationality models formalize resource constraints abstractly in terms of relative Shannon information, namely the Kullback-Leibler Divergence between the agents' prior and posterior policy. Between prior and posterior lies an anytime deliberation process that can be instantiated by sample-based evaluations of the utility function through Markov Chain Monte Carlo (MCMC) optimization. The most simple model assumes a fixed prior and can relate abstract informationtheoretic processing costs to the number of sample evaluations. However, more advanced models would also address the question of learning, that is how the prior is adapted over time such that generated prior proposals become more efficient. In this work we investigate generative neural networks as priors that are optimized concurrently with anytime sample-based decision-making processes such as MCMC. We evaluate this approach on toy examples.

Bounded Rational Decision-Making from Elementary Computations That Reduce Uncertainty

“…In the literature, there are many mechanistic models of decision-making that instantiate decision-making processes with limited resources. Examples include reinforcement learning algorithms with variable depth [ 88 , 89 ], Markov chain Monte Carlo (MCMC) models where only a certain number of samples can be evaluated [ 65 , 85 , 90 ], and evidence accumulation models that accumulate noisy evidence until either a fixed threshold is reached [ 91 , 92 , 93 , 94 , 95 ] or where thresholds are determined dynamically by explicit cost functions depending on the number of allowed evidence accumulation steps [ 96 , 97 ]. Many of these concrete models may be described abstractly by resource parameterizations (Definition 5).…”

“…Due to its analogy with physics, in particular thermodynamics (see, e.g., [ 18 ]), the maximization of Equation ( 30 ) is known as the Free Energy principle of information-theoretic bounded rationality , pioneered in [ 14 , 18 , 62 ], further developed in [ 63 , 64 ], and applied in recent studies of artificial systems, such as generative neural networks trained by Markov chain Monte Carlo methods [ 65 ], or in reinforcement learning as an adaptive regularization strategy [ 66 , 67 ], as well as in recent experimental studies on human behavior [ 68 , 69 ]. Note that there is a formal connection of Equation ( 30 ) and the Free Energy principle of active inference [ 70 ], however, as discussed in [ 64 ] Section 6.3: both Free Energy principles have conceptually different interpretations.…”

An Information-theoretic On-line Learning Principle for Specialization in Hierarchical Decision-Making Systems

“…Our approach belongs to a wider class of models that use information constraints for regularization to deal more efficiently with learning and decision-making problems [11], [31], [28], [25], [19], [35], [26], [18], [15], [3], [21], [38], [20], [17], [16]. One such prominent approach is Trust Region Policy Optimization (TRPO) [39].…”

Specialization in Hierarchical Learning Systems

“…Instead, intelligent agents must invest their limited resources in such a way that they achieve an optimal trade-off between expected utility and resource costs in order to enable efficient learning and acting. This trade-off is the central issue in the fields of bounded or computational rationality with repercussions across other disciplines including economics, psychology, neuroscience and artificial intelligence [3,16,23,24,26,35,55,66,78]. The information-theoretic approach to bounded rationality is a particular instance of bounded rationality where the resource limitations are modeled by information constraints [18,28,54,59,63,64,73,85,92] closely related to Jaynes' maximum entropy or minimum relative entropy principle [41].…”

“…At the heart of information-theoretic models of bounded rationality lies the information utility trade-off for lossy compression, abstraction and hierarchy formation [23]. The optimal utility information trade-off leads to an optimal arrangement of decision-makers and encourages the emergence of specialized agents which in turn facilitates an optimal division of labor reducing computational effort [29,35].…”