Planning and Learning with Stochastic Action Sets

Abstract

In many practical uses of reinforcement learning (RL) the set of actions available at a given state is a random variable, with realizations governed by an exogenous stochastic process. Somewhat surprisingly, the foundations for such sequential decision processes have been unaddressed. In this work, we formalize and investigate MDPs with stochastic action sets (SAS-MDPs) to provide these foundations. We show that optimal policies and value functions in this model have a structure that admits a compact representation. From an RL perspective, we show that Q-learning with sampled action sets is sound. In model-based settings, we consider two important special cases: when individual actions are available with independent probabilities; and a sampling-based model for unknown distributions. We develop poly-time value and policy iteration methods for both cases; and in the first, we offer a poly-time linear programming solution.

Reinforcement Learning When All Actions Are Not Always Available

“…While there is extensive literature on solving sequential decision problems modeled as MDPs (Sutton and Barto 2018), there are few methods designed to handle stochastic action sets. Recently, Boutilier et al (2018) laid the foundation for studying MDPs with stochastic action sets by defining the new SAS-MDP problem formulation, which we review in the background section. After defining SAS-MDPs, Boutilier et al (2018) presented and analyzed the model-based value iteration and policy iteration algorithms and the model-free Q-learning algorithm for SAS-MDPs.…”

“…MDPs and SAS-MDPs (Boutilier et al 2018) are mathematical formulations of sequential decision problems. Before defining SAS-MDPs, we define MDPs.…”

“…As shown by Boutilier et al (2018), one way to model stochastic action sets using the MDP formulation (rather than the SAS-MDP formulation) is to define states such that one can infer A t from S t . Transforming an MDP into a new MDP with A t embedded in S t in this way can result in the size of the state set growing exponentially-by a factor of 2 |B| .…”

“…for all s ∈ S. Similarly, one can extend the Bellman optimality operator (Sutton and Barto 2018): Boutilier et al (2018) showed that the stationary optimal policies exists for SAS-MDPs and can be represented using (state-specific) decision lists (or orderings/rankings) over the action set. As a policy takes into account the available set of actions, an optimal policy chooses the highest ranked action from those that are available.…”

“…As a policy takes into account the available set of actions, an optimal policy chooses the highest ranked action from those that are available. Building upon these results, Boutilier et al (2018) proposed the following update for a tabular estimate, q, of q π * :…”