Guaranteed satisficing and finite regret: Analysis of a cognitive satisficing value function

Abstract

As reinforcement learning algorithms are being applied to increasingly complicated and realistic tasks, it is becoming increasingly difficult to solve such problems within a practical time frame. Hence, we focus on a satisficing strategy that looks for an action whose value is above the aspiration level (analogous to the break-even point), rather than the optimal action. In this paper, we introduce a simple mathematical model called risk-sensitive satisficing (RS) that implements a satisficing strategy by integrating risk-averse and risk-prone attitudes under the greedy policy. We apply the proposed model to the K-armed bandit problems, which constitute the most basic class of reinforcement learning tasks, and prove two propositions. The first is that RS is guaranteed to find an action whose value is above the aspiration level. The second is that the regret (expected loss) of RS is upper bounded by a finite value, given that the aspiration level is set to an "optimal level" so that satisficing implies optimizing. We confirm the results through numerical simulations and compare the performance of RS with that of other representative algorithms for the K-armed bandit problems.

Contextual Exploration Using a Linear Approximation Method Based on Satisficing

“…We call an action whose value exceeds the aspiration level a satisfactory action. RS was originally derived as a generalization of the extremely symmetric form of the conditional probability (see Appendix of [7]). Further, it has the property that the direction is the risk consideration in the value evaluation determined by the action is satisfactory.…”

“…where ℵ opt is the optimal aspiration level, p first is the largest reward distribution, and p second is the second largest reward distribution. RS can find the optimal action sequence with a small number of exploratory actions by setting the aspiration level in accordance with equation (4) [7]. This is because, when given an optimal aspiration level, the agent can only be satisfied with the action with the largest reward distribution (the optimal action).…”

“…From the theory of linear regression, the update of the estimated parameter θ of the action value is defined by Equation (7).…”

“…Furthermore, the satisficing algorithm does not preclude optimization. Rather, by setting the optimal aspiration level, it can aim at finding optimal actions as more surely than optimization algorithms [7]. Whether to seek optimal actions or satisficing actions depends on the goal of the task.…”

“…A previous study related to satisficing has proposed risk-sensitive satisficing (RS) [6], which introduces satisficing into the value function in reinforcement learning. RS can perform efficient learning with a small amount of exploration, though it depends on the target level [7]. However, functional and linear approximation methods for RS have not been proposed; it is not easy to apply RS to deep reinforcement learning because it has no mechanism for feature-based learning.…”