Online Stacking Using RL with Positional and Tactical Features

Abstract

We study the scenario where some items are stored temporarily in stacks and where it is not allowed to put an item on top of another item leaving earlier. An arriving item is assigned to a stack based only on information on the arrival and departure times for the new item and items currently stored. The objective is to minimize the maximum number of stacks used over time. This problem is referred to as online stacking. We use Reinforcement Learning (RL) techniques to improve heuristics earlier presented in the literature. Using an analogy to chess, we look at positional and tactical features where the former give high priority to stacking configurations that are well suited to meet the challenges on a long-term basis and the latter focus on using few stacks on a short-term basis. We show how the RL approach finds the optimal mix of positional and tactical features to be used at different stages of the stacking process. We document quantitatively that positional features play a bigger role at stages of the stacking process with few items stored. We believe that the RL approach combining positional and tactical features can be used in many other online settings within operations research.

An asymptotically optimal algorithm for online stacking

“…Simple heuristics for online stacking similar to Algorithm Online have been presented by Borgman et al [4], Duinkerken et al [8], Hamdi et al [9], and Wang et al [18] without providing a proof of asymptotic optimality. Olsen shows in [12] how Reinforcement Learning can be used to improve simple online stacking heuristics. Finally, we mention the work of Rei and Pedroso [16] and König et al [11] on related problems within the steel industry as well as the PhD thesis by Pacino [14] on container ship stowage.…”

Oblivious Stacking and MAX k-CUT for Circle Graphs

“…Olsen and Gross [14] have developed a polynomial time algorithm for online coloring with a competitive ratio that converges to 1 in probability if the endpoints of the storage time intervals are picked independently and uniformly at random. Olsen [13] has also shown how to use Reinforcement Learning to improve online stacking heuristics.…”