Trading Value and Information in MDPs

Rubin, Jonathan E.; Shamir, Ohad; Tishby, Naftali

doi:10.1007/978-3-642-24647-0_3

Cited by 68 publications

(92 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The theory was originally developed in applied settings for the development of telecommunications and cryptography systems [16,18]. However, it has seen wide application in fields related to cognitive science, including theoretical neuroscience [19], statistical complexity theory [20,21], and models of rational action under information processing constraints [22,23]. Fruitful applications of information theory to language are now possible due to large datasets and computational power that make it possible to estimate information-theoretic quantities such as entropy from language data [24].…”

Section: What Is Communicative Efficiency?mentioning

confidence: 99%

How Efficiency Shapes Human Language

Gibson

Futrell

Piantadosi

et al. 2019

Trends in Cognitive Sciences

298

196

View full text Add to dashboard Cite

Cognitive science applies diverse tools and perspectives to study human language. Recently, an exciting body of work has examined linguistic phenomena through the lens of efficiency in usage: what otherwise puzzling features of language find explanation in formal accounts of how language might be optimized for communication and learning? Here, we review studies that deploy formal tools from probability and information theory to understand how and why language works the way that it does, focusing on phenomena ranging from the lexicon through syntax. These studies show how a pervasive pressure for efficiency guides the forms of natural language and indicate that a rich future for language research lies in connecting linguistics to cognitive psychology and mathematical theories of communication and inference.

show abstract

Section: What Is Communicative Efficiency?mentioning

confidence: 99%

How Efficiency Shapes Human Language

Gibson

Futrell

Piantadosi

et al. 2019

Trends in Cognitive Sciences

298

196

View full text Add to dashboard Cite

show abstract

“…Following the work of (Simon, 1972) decision-making with limited information-processing resources has been studied extensively in psychology, economics, political science, industrial organization, computer science, and artificial intelligence research. In this paper, we use an information-theoretic model of decisionmaking under resource constraints (McKelvey and Palfrey, 1995;Kappen, 2005;Wolpert, 2006;Todorov, 2009;Peters et al, 2010;Theodorou et al, 2010;Rubin et al, 2012). In particular, Braun et al (2011) and Ortega and Braun (2011) present a framework in which gain in expected utility is traded off against the adaptation cost of changing from an initial behavior to a posterior behavior.…”

Section: Discussionmentioning

confidence: 99%

“…In this study, we use an information-theoretic model of bounded rational decision-making Braun, 2012, 2013;Braun and Ortega, 2014; that has precursors in the economic literature (McKelvey and Palfrey, 1995;Mattsson and Weibull, 2002;Sims, 2003Sims, , 2005Sims, , 2006Sims, , 2010Wolpert, 2006) and that is closely related to recent advances in the information theory of perception-action systems (Todorov, 2007(Todorov, , 2009Still, 2009;Friston, 2010;Peters et al, 2010;Tishby and Polani, 2011;Daniel et al, 2012Daniel et al, , 2013Kappen et al, 2012;Rawlik et al, 2012;Rubin et al, 2012;Neymotin et al, 2013;Tkačik and Bialek, 2014;Palmer et al, 2015). The basis of this approach is formalized by a free energy principle that trades off expected utility, and the cost of computation that is required to adapt the system accordingly in order to achieve high utility.…”

Section: Introductionmentioning

confidence: 99%

Bounded Rationality, Abstraction, and Hierarchical Decision-Making: An Information-Theoretic Optimality Principle

et al. 2015

View full text Add to dashboard Cite

Abstraction and hierarchical information processing are hallmarks of human and animal intelligence underlying the unrivaled flexibility of behavior in biological systems. Achieving such flexibility in artificial systems is challenging, even with more and more computational power. Here, we investigate the hypothesis that abstraction and hierarchical information processing might in fact be the consequence of limitations in information-processing power. In particular, we study an information-theoretic framework of bounded rational decision-making that trades off utility maximization against information-processing costs. We apply the basic principle of this framework to perception-action systems with multiple information-processing nodes and derive bounded-optimal solutions. We show how the formation of abstractions and decision-making hierarchies depends on information-processing costs. We illustrate the theoretical ideas with example simulations and conclude by formalizing a mathematically unifying optimization principle that could potentially be extended to more complex systems.

show abstract

“…A recent theory has suggested that random choice behavior results from computational limitations on the policy [ 46 ]. In this framework, optimal policy is assumed to be a tradeoff between expected reward and expected information cost.…”

Section: Discussionmentioning

confidence: 99%

Heterogeneous Suppression of Sequential Effects in Random Sequence Generation, but Not in Operant Learning

Shteingart

Loewenstein

2016

PLoS ONE

View full text Add to dashboard Cite

There is a long history of experiments in which participants are instructed to generate a long sequence of binary random numbers. The scope of this line of research has shifted over the years from identifying the basic psychological principles and/or the heuristics that lead to deviations from randomness, to one of predicting future choices. In this paper, we used generalized linear regression and the framework of Reinforcement Learning in order to address both points. In particular, we used logistic regression analysis in order to characterize the temporal sequence of participants’ choices. Surprisingly, a population analysis indicated that the contribution of the most recent trial has only a weak effect on behavior, compared to more preceding trials, a result that seems irreconcilable with standard sequential effects that decay monotonously with the delay. However, when considering each participant separately, we found that the magnitudes of the sequential effect are a monotonous decreasing function of the delay, yet these individual sequential effects are largely averaged out in a population analysis because of heterogeneity. The substantial behavioral heterogeneity in this task is further demonstrated quantitatively by considering the predictive power of the model. We show that a heterogeneous model of sequential dependencies captures the structure available in random sequence generation. Finally, we show that the results of the logistic regression analysis can be interpreted in the framework of reinforcement learning, allowing us to compare the sequential effects in the random sequence generation task to those in an operant learning task. We show that in contrast to the random sequence generation task, sequential effects in operant learning are far more homogenous across the population. These results suggest that in the random sequence generation task, different participants adopt different cognitive strategies to suppress sequential dependencies when generating the “random” sequences.

show abstract

Trading Value and Information in MDPs

Cited by 68 publications

References 11 publications

How Efficiency Shapes Human Language

How Efficiency Shapes Human Language

Bounded Rationality, Abstraction, and Hierarchical Decision-Making: An Information-Theoretic Optimality Principle

Heterogeneous Suppression of Sequential Effects in Random Sequence Generation, but Not in Operant Learning

Contact Info

Product

Resources

About