Equi-Reward Utility Maximizing Design in Stochastic Environments

Keren, Sarah; Pineda, Luis; Gal, Avigdor; Karpas, Erez; Zilberstein, Shlomo

doi:10.24963/ijcai.2017/608

Cited by 17 publications

(30 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We present an actor-designer framework to mitigate the impacts of NSE by shaping the environment. The general idea of environment modification to influence the behaviors of the acting agents has been previously explored in other contexts (Zhang, Chen, and Parkes 2009), such as to accelerate agent learning (Randløv 2000), to quickly infer the goals of the actor (Keren, Gal, and Karpas 2014), and to maximize the agent's reward (Keren et al 2017). We study how environment shaping can mitigate NSE.…”

Section: Discussion and Future Workmentioning

confidence: 99%

Mitigating Negative Side Effects via Environment Shaping

Saisubramanian¹,

Zilberstein²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Agents operating in unstructured environments often produce negative side effects (NSE), which are difficult to identify at design time. While the agent can learn to mitigate the side effects from human feedback, such feedback is often expensive and the rate of learning is sensitive to the agent's state representation. We examine how humans can assist an agent, beyond providing feedback, and exploit their broader scope of knowledge to mitigate the impacts of NSE. We formulate this problem as a human-agent team with decoupled objectives. The agent optimizes its assigned task, during which its actions may produce NSE. The human shapes the environment through minor reconfiguration actions so as to mitigate the impacts of the agent's side effects, without affecting the agent's ability to complete its assigned task. We present an algorithm to solve this problem and analyze its theoretical properties. Through experiments with human subjects, we assess the willingness of users to perform minor environment modifications to mitigate the impacts of NSE. Empirical evaluation of our approach shows that the proposed framework can successfully mitigate NSE, without affecting the agent's ability to complete its assigned task.

show abstract

Section: Discussion and Future Workmentioning

confidence: 99%

Mitigating Negative Side Effects via Environment Shaping

Saisubramanian¹,

Zilberstein²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Instead, the possibility to strategically act on the environmental dynamics is studied in a limited number of works only. Some approaches belonging to the planning area [12,38], some are constrained to specific forms of environment configurability [8,9,34], and others based on the curriculum learning framework [4,7]. The goal of the dissertation [18] is to provide a uniform treatment of environment configurability in its diverse aspects.…”

Section: Configurable Environmentsmentioning

confidence: 99%

Configurable Environments in Reinforcement Learning: An Overview

Metelli

2022

Special Topics in Information Technology

View full text Add to dashboard Cite

Reinforcement Learning (RL) has emerged as an effective approach to address a variety of complex control tasks. In a typical RL problem, an agent interacts with the environment by perceiving observations and performing actions, with the ultimate goal of maximizing the cumulative reward. In the traditional formulation, the environment is assumed to be a fixed entity that cannot be externally controlled. However, there exist several real-world scenarios in which the environment offers the opportunity to configure some of its parameters, with diverse effects on the agent’s learning process. In this contribution, we provide an overview of the main aspects of environment configurability. We start by introducing the formalism of the Configurable Markov Decision Processes (Conf-MDPs) and we illustrate the solutions concepts. Then, we revise the algorithms for solving the learning problem in Conf-MDPs. Finally, we present two applications of Conf-MDPs: policy space identification and control frequency adaptation.

show abstract

“…SWOPP relates to the environment design problem in AI in which an interested party, the system designer, can make changes to the environment of an agent in order to encourage desired behavior [ 14 ]. Keren et al [ 15 ] model environment design settings in which the system designer and the worker optimize the same utility function. The designer attempts to introduce changes to the environment that would reduce the cost of achieving the joint goals.…”

Section: Related Workmentioning

confidence: 99%

Supervisor-Worker Problems with an Application in Education

Shmaryahu

Gal

Shani

2021

Sensors

View full text Add to dashboard Cite

In many e-learning settings, allowing students to choose which skills to practice encourages their motivation and contributes to learning. However, when given choice, students may prefer to practice skills that they already master, rather than practice skills they need to master. On the other hand, requiring students only to practice their required skills may reduce their motivation and lead to dropout. In this paper, we model this tradeoff as a multi-agent planning task, which we call SWOPP (Supervisor- Worker Problem with Partially Overlapping goals), involving two agents—a supervisor (teacher) and a worker (student)—each with different, yet non-conflicting, goals. The supervisor and worker share joint goals (mastering skills). The worker plans to achieve his/her own goals (completing an e-learning session) at a minimal cost (effort required to solve problems). The supervisor guides the worker towards achieving the joint goals by controlling the problems in the choice set for the worker. We provide a formal model for the SWOPP task and two sound and complete algorithms for the supervisor to guide the worker’s plan to achieve their joint goals. We deploy SWOPP for the first time in a real-world study to personalize math questions for K5 students using an e-learning software in schools. We show that SWOPP was able to guide students’ interactions with the software to practice necessary skills without deterring their motivation.

show abstract

Equi-Reward Utility Maximizing Design in Stochastic Environments

Cited by 17 publications

References 7 publications

Mitigating Negative Side Effects via Environment Shaping

Mitigating Negative Side Effects via Environment Shaping

Configurable Environments in Reinforcement Learning: An Overview

Supervisor-Worker Problems with an Application in Education

Contact Info

Product

Resources

About