Can Deep Reinforcement Learning Improve Inventory Management? Performance on Lost Sales, Dual-Sourcing, and Multi-Echelon Problems

Abstract

Problem definition: Is deep reinforcement learning (DRL) effective at solving inventory problems? Academic/practical relevance: Given that DRL has successfully been applied in computer games and robotics, supply chain researchers and companies are interested in its potential in inventory management. We provide a rigorous performance evaluation of DRL in three classic and intractable inventory problems: lost sales, dual sourcing, and multi-echelon inventory management. Methodology: We model each inventory problem as a Markov decision process and apply and tune the Asynchronous Advantage Actor-Critic (A3C) DRL algorithm for a variety of parameter settings. Results: We demonstrate that the A3C algorithm can match the performance of the state-of-the-art heuristics and other approximate dynamic programming methods. Although the initial tuning was computationally demanding and time demanding, only small changes to the tuning parameters were needed for the other studied problems. Managerial implications: Our study provides evidence that DRL can effectively solve stationary inventory problems. This is especially promising when problem-dependent heuristics are lacking. Yet, generating structural policy insight or designing specialized policies that are (ideally provably) near optimal remains desirable.

Solving Inventory Management Problems with Inventory-dynamics-informed Neural Networks

“…To the best of our knowledge, ours is the first approach that shows how neural networks can compete with and even outperform inventory-management approaches developed after years of specialized research on dual sourcing. From a methodological perspective, our paper is close to Gijsbrechts et al (2020), who apply RL-based policy learning to a variety of inventory management problems. In particular, the authors show that the presented Asynchronous Advantage Actor-Critic (A3C) algorithm competes favorably with several known heuristics for lost-sales, dual-sourcing and multi-echelon models.…”

“…Theoretically, neural networks are able to represent such policies under mild conditions, as specified by universal approximation theorems (Hornik 1991, Hanin and Sellke 2017, Park et al 2020. In practice, however, designing neural networks that represent optimal policies of discrete-time stochastic dynamical systems has been challenging (Gijsbrechts et al 2020, Boute et al 2021b. We next outline an algorithm that does so efficiently.…”

“…Using state-of-the-art machine learning methods combined with domain-specific knowledge is a relatively new and promising venue for inventory management research (Xin andVan Mieghem 2021, Song et al 2020). An important contribution in this direction is the work by Gijsbrechts et al (2020), who utilize deep reinforcement learning (RL)-based optimization to control three archetypal inventory management problems. The authors show that an actor-critic RL algorithm can deliver competitive performance against problem-specific heuristics.…”

Multi-echelon Supply Chains with Uncertain Seasonal Demands and Lead Times Using Deep Reinforcement Learning

“…We now present related works that handle the problem using an order-based approach. Gijsbrechts et al (2019) propose a proof of concept by using Deep-RL on three different problems: dual-sourcing or dual-mode, lost sales, and multi-echelon inventory models. In the multi-echelon setup, demands are uncertain and regular, and lead times are determin-istic.…”

“…Some works in the literature use Deep RL on related problems, but they usually deal with smaller supply chain networks, with two-echelon or serial supply chains (Oroojlooyjadid 2019;Kemmer et al 2018;Hutse 2019;Gijsbrechts et al 2019;Peng et al 2019). Considering serial supply chains, the dimensionality of the action space is close to the number of echelons.…”