Reinforcement Learning Your Way: Agent Characterization through Policy Regularization

Abstract

The increased complexity of state-of-the-art reinforcement learning (RL) algorithms has resulted in an opacity that inhibits explainability and understanding. This has led to the development of several post hoc explainability methods that aim to extract information from learned policies, thus aiding explainability. These methods rely on empirical observations of the policy, and thus aim to generalize a characterization of agents’ behaviour. In this study, we have instead developed a method to imbue agents’ policies with a characteristic behaviour through regularization of their objective functions. Our method guides the agents’ behaviour during learning, which results in an intrinsic characterization; it connects the learning process with model explanation. We provide a formal argument and empirical evidence for the viability of our method. In future work, we intend to employ it to develop agents that optimize individual financial customers’ investment portfolios based on their spending personalities.

Can Interpretable Reinforcement Learning Manage Prosperity Your Way?

“…Interpretation of RL agents typically follows model training [26,27,28]; our ambition is to impose a desired characteristic behaviour during training, thus making it an intrinsic property of the agent. Based on a prior that defines a desired behaviour, we extend the deep deterministic policy gradient (DDPG [29]) objective function with a regularisation term [15]. Formally, for each agent i, this objective function is given by:…”

“…We have previously investigated the interpretability of systems of multiple RL agents [15]. A regularisation term in the objective function imposed a desired agent behaviour during training.…”

Reinforcement Learning with Intrinsic Affinity for Personalized Prosperity Management

“…Policy regularization has been shown to be helpful and never detrimental to convergence [21]. Although most policy regularization methods aim to improve learning performance, they can also control the learning process and imbue the policy with an intrinsic behavior [22]. Here, the objective function is regularized with a predefined prior action distribution that defines a desirable characteristic:…”

“…These agents learned to either prefer left turns, right turns, or to avoid going straight by taking a zig-zag approach to their destination. In contrast to constrained RL, which avoids certain states, the policy regularization in [22] encourages certain actions irrespective of the state and is a new direction for RL.…”

Can Interpretable Reinforcement Learning Manage Prosperity Your Way?

“…We distinguish between explainability and interpretability: explainability refers to a symbolic representation of the knowledge a model has learned, while interpretability is necessary for reasoning about a model's predictions. We have previously investigated the interpretability of systems of multiple RL agents [15]: a regularisation term in the objective function instilled a desired agent behaviour during training. For our current purpose of prosperity management, we create prototypical RL agents which have intrinsic affinities for certain asset classes.…”