Building a Foundation for Data-Driven, Interpretable, and Robust Policy Design using the AI Economist

Trott, Alexander; Srinivasa, Sunil; Wal, Douwe van der; Haneuse, Sebastien; Zheng, Stephan

doi:10.48550/arxiv.2108.02904

Cited by 6 publications

(7 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on the present discussion, future work might consider mechanism design using agents that are themselves engaged in active inference (e.g., within a general equilibrium macroeconomic model that is being utilized to understand a country's SWB (Hill et al, 2021)). Parallel work in the RL literature has developed similar large-scale simulations to develop dynamic taxation and subsidy policies that consider multiple objectives, policy levers, and behavioral responses from strategic actors that optimize for their individual objectives (Trott et al, 2021). A taxation policy from such reinforcement learning simulations can even outperform optimal static policies in terms of productivity and equity (Zheng et al, 2021).…”

Section: Target Outcomes Of Interventionsmentioning

confidence: 99%

A computational neuroscience perspective on subjective wellbeing within the active inference framework

Smith¹,

Varshney²,

Nagayama³

et al. 2022

Intnl. J. Wellbeing

View full text Add to dashboard Cite

Understanding and promoting subjective wellbeing (SWB) has been the topic of increasing research, due in part to its potential contributions to health and productivity. To date, the conceptualization of SWB has been grounded within social psychology and largely focused on self-report measures. In this paper, we explore the potentially complementary tools and theoretical perspectives offered by computational neuroscience, with a focus on the active inference (AI) framework. This framework is motivated by the fact that the brain does not have direct access to the world; to select actions, it must instead infer the most likely external causes of the sensory input it receives from both the body and the external world. Because sensory input is always consistent with multiple interpretations, the brain’s internal model must use background knowledge, in the form of prior expectations, to make a “best guess” about the situation it is in and how it will change by taking one action or another. This best guess arises by minimizing an error signal representing the deviation between predicted and observed sensations given a chosen action—quantified mathematically by a variable called free energy (FE). Crucially, recent proposals have illustrated how emotional experience may emerge within AI as a natural consequence of the brain keeping track of the success of its model in selecting actions to minimize FE. In this paper, we draw on the concepts and mathematics in AI to highlight how different computational strategies can be used to minimize FE—some more successfully than others. This affords a characterization of how diverse individuals may adopt unique strategies for achieving high SWB. It also highlights novel ways in which SWB could be effectively improved. These considerations lead us to propose a novel computational framework for understanding SWB. We highlight several parameters in these models that could explain individual and cultural differences in SWB, and how they might inspire novel interventions. We conclude by proposing a line of future empirical research based on computational modelling that could complement current approaches to the study of wellbeing and its improvement.

show abstract

Section: Target Outcomes Of Interventionsmentioning

confidence: 99%

A computational neuroscience perspective on subjective wellbeing within the active inference framework

Smith¹,

Varshney²,

Nagayama³

et al. 2022

Intnl. J. Wellbeing

View full text Add to dashboard Cite

show abstract

“…The COVID environment, developed by Kompella et al (2020), simulates a population using the SEIR model of individual infection dynamics. The RL policymaker adjusts the severity of social distancing regulations while balancing economic health (better with lower regulations) and public health (better with higher regulations), similar in spirit to Trott et al (2021). The population attributes (proportion of adults, number of hospitals) and infection dynamics (random testing rate, infection rate) are based on data from Austin, Texas.…”

Section: Environmentsmentioning

confidence: 99%

“…Reward hacking, or the gaming of misspecified reward functions by RL agents, has appeared in a variety of contexts, such as game playing (Ibarz et al, 2018), text summarization (Paulus et al, 2018), and autonomous driving (Knox et al, 2021). These examples show that better algorithms and models are not enough; for human-centered applications such as healthcare (Yu et al, 2019), economics (Trott et al, 2021) and robotics (Kober et al, 2013), RL algorithms must be safe and aligned with human objectives (Bommasani et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

Pan¹,

Bhatia²,

Steinhardt³

2022

Preprint

View full text Add to dashboard Cite

Reward hacking-where RL agents exploit gaps in misspecified reward functions-has been widely observed, but not yet systematically studied. To understand how reward hacking arises, we construct four RL environments with misspecified rewards. We investigate reward hacking as a function of agent capabilities: model capacity, action space resolution, observation space noise, and training time. More capable agents often exploit reward misspecifications, achieving higher proxy reward and lower true reward than less capable agents. Moreover, we find instances of phase transitions: capability thresholds at which the agent's behavior qualitatively shifts, leading to a sharp decrease in the true reward. Such phase transitions pose challenges to monitoring the safety of ML systems. To address this, we propose an anomaly detection task for aberrant policies and offer several baseline detectors.

show abstract

“…Deep reinforcement learning (RL) is a powerful learning framework to train AI agents. RL agents have beaten humans at several strategy games (1,2), trained robotic arms (3), and been used to design economic policies (4,5).…”

Section: Introductionmentioning

confidence: 99%

WarpDrive: Extremely Fast End-to-End Deep Multi-Agent Reinforcement Learning on a GPU

Lan,

Srinivasa,

Wang

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Deep reinforcement learning (RL) is a powerful framework to train decision-making models in complex dynamical environments. However, RL can be slow as it learns through repeated interaction with a simulation of the environment. Accelerating RL requires both algorithmic and engineering innovations. In particular, there are key systems engineering bottlenecks when using RL in complex environments that feature multiple agents or highdimensional state, observation, or action spaces, for example. We present WarpDrive, a flexible, lightweight, and easy-to-use open-source RL framework that implements end-toend multi-agent RL on a single GPU (Graphics Processing Unit), building on PyCUDA and PyTorch. Using the extreme parallelization capability of GPUs, WarpDrive enables ordersof-magnitude faster RL compared to common implementations that blend CPU simulations and GPU models. Our design runs simulations and the agents in each simulation in parallel. It eliminates data copying between CPU and GPU. It also uses a single simulation data store on the GPU that is safely updated in-place. Together, this allows the user to run thousands of concurrent multi-agent simulations and train on extremely large batches of experience. For example, WarpDrive yields 2.9 million environment steps/second with 2000 environments and 1000 agents (at least 100× higher throughput compared to a CPU implementation) in a benchmark Tag simulation. WarpDrive provides a lightweight Python interface and environment wrappers to simplify usage and promote flexibility and extensions. As such, WarpDrive provides a framework for building high-throughput RL systems. *: TL and SS contributed equally. TL and SS designed and developed WarpDrive. TL built the core CUDA library, including the DataManager and FunctionManagers. SS built the environment wrapper and the training pipeline. SS and TL wrote the simulation examples and unit tests. SS and TL ran RL experiments. SZ, SS, and TL drafted the paper. SZ conceived and directed the project. Code is available at https://www.github.com/ salesforce/warp-drive. We thank Alexander Trott for valuable comments on this paper.The name WarpDrive is inspired by the science fiction concept of a fictional superluminal spacecraft propulsion system. Moreover, at the time of writing, a warp is a group of 32 threads that are executing at the same time in (certain) GPUs.

show abstract

Building a Foundation for Data-Driven, Interpretable, and Robust Policy Design using the AI Economist

Cited by 6 publications

References 26 publications

A computational neuroscience perspective on subjective wellbeing within the active inference framework

A computational neuroscience perspective on subjective wellbeing within the active inference framework

The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

WarpDrive: Extremely Fast End-to-End Deep Multi-Agent Reinforcement Learning on a GPU

Contact Info

Product

Resources

About