Possibility Before Utility: Learning And Using Hierarchical Affordances

Abstract

Reinforcement learning algorithms struggle on tasks with complex hierarchical dependency structures. Humans and other intelligent agents do not waste time assessing the utility of every high-level action in existence, but instead only consider ones they deem possible in the first place. By focusing only on what is feasible, or "afforded", at the present moment, an agent can spend more time both evaluating the utility of and acting on what matters. To this end, we present Hierarchical Affordance Learning (HAL), a method that learns a model of hierarchical affordances in order to prune impossible subtasks for more effective learning. Existing works in hierarchical reinforcement learning provide agents with structural representations of subtasks but are not affordance-aware, and by grounding our definition of hierarchical affordances in the present state, our approach is more flexible than the multitude of approaches that ground their subtask dependencies in a symbolic history. While these logic-based methods often require complete knowledge of the subtask hierarchy, our approach is able to utilize incomplete and varying symbolic specifications. Furthermore, we demonstrate that relative to non-affordance-aware methods, HAL agents are better able to efficiently learn complex tasks, navigate environment stochasticity, and acquire diverse skills in the absence of extrinsic supervision-all of which are hallmarks of human learning. 1

Reducing redundancy of skills in hierarchical reinforcement learning with offline data

“…The purpose of this experiment is to validate the effectiveness of our algorithm in addressing the problem of decision difficulty caused by skill redundancy in long-sequence composite tasks. To achieve this, we plan to compare our algorithm with two prominent methods in hierarchical reinforcement learning: the classic Option-Critic algorithm and the HAL algorithm 16 , which has shown good performance in sparse reward tasks.…”