The QLBS Q-Learner Goes NuQLear: Fitted Q Iteration, Inverse RL, and Option Portfolios

Abstract

The QLBS model is a discrete-time option hedging and pricing model that is based on Dynamic Programming (DP) and Reinforcement Learning (RL). It combines the famous Q-Learning method for RL with the Black-Scholes (-Merton) model's idea of reducing the problem of option pricing and hedging to the problem of optimal rebalancing of a dynamic replicating portfolio for the option, which is made of a stock and cash. Here we expand on several NuQLear (Numerical Q-Learning) topics with the QLBS model. First, we investigate the performance of Fitted Q Iteration for a RL (data-driven) solution to the model, and benchmark it versus a DP (modelbased) solution, as well as versus the BSM model. Second, we develop an Inverse Reinforcement Learning (IRL) setting for the model, where we only observe prices and actions (re-hedges) taken by a trader, but not rewards. Third, we outline how the QLBS model can be used for pricing portfolios of options, rather than a single option in isolation, thus providing its own, data-driven and model independent solution to the (in)famous volatility smile problem of the Black-Scholes model.I would like to thank Eric Berger and Vivek Kapoor for stimulating discussions. I thank Bohui Xi, Tianrui Zhao, and Yuhan Liu for an initial implementation of a DP solution of the QLBS model.

Market Self-Learning of Signals, Impact and Optimal Trading: Invisible Hand Inference with Free Energy (Or, How We Learned to Stop Worrying and Love Bounded Rationality)

“…Quadratic risk-adjusted objective functions were considered in an apparently different problem of optimal option pricing and hedging using a model-free, data-driven approach in the work by one of the authors [26,27]. The approach used in this work assumes off-line, batch-mode learning, that enables using data-efficient batch RL methods such as Fitted Q Iteration [15,37].…”

“…Note that the multi-period portfolio optimization problem (24) assumes that an optimal policy that determines actions a t is a deterministic policy that can also be described as a delta-like probability distribution π(a t |y t ) = δ (a t − a t (y t )) (27) where the optimal deterministic action a t (y t ) is obtained by maximization of the objective (24) with respect to controls a t .…”

“…But the actual trading data may be sub-optimal, or noisy at times, because of model misspecifications, market timing lags, human errors etc. Potential presence of such sub-optimal actions in data poses serious challenges, if we try to assume deterministic policy (27) that assumes the the action chosen is always an optimal action. This is because such events should have zero probability under these model assumptions, and thus would produced vanishing path probabilities if observed in data.…”

“…Instead of assuming a deterministic policy (27), stochastic policies described by smoothed distributions π(a t |y t ), are more useful for inverse problems such as the problem of inverse portfolio optimization. In this approach, instead of maximization with respect to deterministic policy/action a t , we re-formulate the problem as maximization over probability distributions π(a t |y t ):…”

Neural Network for Pricing and Universal Static Hedging of Contingent Claims

“…They have been used in Liu et al [2019b], Liu et al [2019a], Ackerer et al [2019] and Bayer et al [2019] for calibration of stochastic volatility and rough stochastic volatility models. Another strand of literature is on the applications of deep reinforcement learning, specifically related to data driven pricing and hedging of portfolios, which includes notably the work of Buehler et al [2019] and Halperin [2019]. Application of neural networks for model-based pricing of early exercise options includes the approach of policy iteration in Becker et al [2019], value function iteration in Haugh and Kogan [2004], Kohler et al [2010] and Lapeyre and Lelong [2019].…”