A Hybrid Reinforcement Learning Algorithm for 2D Irregular Packing Problems

Abstract

Packing problems, also known as nesting problems or bin packing problems, are classic and popular NP-hard problems with high computational complexity. Inspired by classic reinforcement learning (RL), we established a mathematical model for two-dimensional (2D) irregular-piece packing combined with characteristics of 2D irregular pieces. An RL algorithm based on Monte Carlo learning (MC), Q-learning, and Sarsa-learning is proposed in this paper to solve a 2D irregular-piece packing problem. Additionally, mechanisms of reward–return and strategy-update based on piece packing were designed. Finally, the standard test case of irregular pieces was used for experimental testing to analyze the optimization effect of the algorithm. The experimental results show that the proposed algorithm can successfully realize packing of 2D irregular pieces. A similar or better optimization effect can be obtained compared to some classical heuristic algorithms. The proposed algorithm is an early attempt to use machine learning to solve 2D irregular packing problems. On the one hand, our hybrid RL algorithm can provide a basis for subsequent deep reinforcement learning (DRL) to solve packing problems, which has far-reaching theoretical significance. On the other hand, it has practical significance for improving the utilization rate of raw materials and broadening the application field of machine learning.

Learning Based 2D Irregular Shape Packing

“…Learning irregular packing skills is a much more challenging problem, where the learning model has to recognize the complex object shapes and handle the continuous decision space of object poses. Two recent works [Fang et al 2023;Goyal and Deng 2020] considered packing of irregular shapes. Goyal and Deng [2020] proposed a problem set for 3D packing problems and showed that a neural shape selector trained via reinforcement learning outperforms heuristic baselines.…”

“…Goyal and Deng [2020] proposed a problem set for 3D packing problems and showed that a neural shape selector trained via reinforcement learning outperforms heuristic baselines. Fang et al [2023] also trained a neural shape selector via reinforcement learning. Both algorithms use heuristic rules for the low-level pose computation and Fang et al [2023] even used NFP-based pose computation.…”

“…Fang et al [2023] also trained a neural shape selector via reinforcement learning. Both algorithms use heuristic rules for the low-level pose computation and Fang et al [2023] even used NFP-based pose computation. However, only learning the high-level selector is not enough for UV packing, since low-level pose selection can still be a major computational bottleneck.…”

“…Having observed 𝑠 𝑖 , our LPN can be represented as a policy function 𝑎 𝑖 = 𝜋 LPN (𝑠 𝑖 ) that maps the state to an action 𝑎 𝑖 in the action space 𝐴. Unlike prior works for learning-based regular shape packing [Zhao et al 2021] or shape ordering [Fang et al 2023], the design of action space 𝐴 for irregular packing is much more challenging. On the one hand, a valid action space should only consist of collision-free patch poses, while identifying these actions can involve extensive collision checks.…”

A deep reinforcement learning algorithm for the rectangular strip packing problem

“…Xiaofei Xu et al [1] tried to solve the rectangular packing problem by using transfer ant colony reinforcement learning, and with the help of the "trial and error" learning mode, the acquisition and update of the knowledge were completed by using an ant colony with self-learning ability in the knowledge matrix. J Fang et al [37,38] and X Zhao et al [39] used the Monte Carlo (MC) algorithm and Q-learning algorithm in reinforcement learning to solve the sequence optimization problem in 2D irregular piece packing and rectangular packing, respectively, but the search had a certain randomness. The combination of machine learning and intelligent algorithms has also achieved certain achievements in some fields.…”