OR-Gym: A Reinforcement Learning Library for Operations Research Problems

Abstract

Reinforcement learning (RL) has been widely applied to game-playing and surpassed the best human-level performance in many domains, yet there are few use-cases in industrial or commercial settings. We introduce OR-Gym, an open-source library for developing reinforcement learning algorithms to address operations research problems. In this paper, we apply reinforcement learning to the knapsack, multi-dimensional bin packing, multi-echelon supply chain, and multi-period asset allocation model problems, as well as benchmark the RL solutions against MILP and heuristic models. These problems are used in logistics, finance, engineering, and are common in many business operation settings. We develop environments based on prototypical models in the literature and implement various optimization and heuristic models in order to benchmark the RL results. By re-framing a series of classic optimization problems as RL tasks, we seek to provide a new tool for the operations research community, while also opening those in the RL community to many of the problems and challenges in the OR field.

Towards Automatic Actor-Critic Solutions to Continuous Control

“…The scores for the InvManagement-v1 and Newsvendor-v0 environments are listed in Tables 2 and 3, respectively. AAC outperforms our baselines as well as the RL results reported in [22].…”

“…Next we consider two inventory management problems (IMPs) proposed by [22] and [3]. IMPs involve managing a supply chain to meet customer demand while balancing costs associated with ordering and carrying new materials 3 .…”

“…Rand-SAC, SAC and AAC collect 500, 000 steps of experience. The RL result from[22] uses Proximal Policy Optimization (PPO)[42]. The oracle method generates a theoretical upper bound by solving an optimization problem with information not available to the agent.…”

“…We refer the reader to the original references[22] &[3] for thorough descriptions of the environments 4. We make two changes to the DMC experiment settings before launch: kmax is lowered to 5 from 15 because these environments have short time horizons of 40 and 30, and the epoch length is lowered to 500 from 1000 because we are testing after 500, 000 total timesteps.…”

Towards Standardizing Reinforcement Learning Approaches for Stochastic Production Scheduling

“…First steps in the direction of RL project standardization were taken by OpenAI Gym, whereby a general RL application programming interface (API) is defined. Through [9], OpenAI Gym simulations for varied combinatorial optimization problems from the field of OR, e.g. the bin packing or the traveling salesman problem, are made available.…”