Learning Practically Feasible Policies for Online 3D Bin Packing

Abstract

We tackle the Online 3D Bin Packing Problem, a challenging yet practically useful variant of the classical Bin Packing Problem. In this problem, the items are delivered to the agent without informing the full sequence information. Agent must directly pack these items into the target bin stably without changing their arrival order, and no further adjustment is permitted. Online 3D-BPP can be naturally formulated as Markov Decision Process (MDP). We adopt deep reinforcement learning, in particular, the on-policy actorcritic framework, to solve this MDP with constrained action space. To learn a practically feasible packing policy, we propose three critical designs. First, we propose an online analysis of packing stability based on a novel stacking tree. It attains a high analysis accuracy while reducing the computational complexity from O(N 2 ) to O(N log N ), making it especially suited for RL training. Second, we propose a decoupled packing policy learning for different dimensions of placement which enables high-resolution spatial discretization and hence high packing precision. Third, we introduce a reward function that dictates the robot to place items in a far-to-near order and therefore simplifies the collision avoidance in movement planning of the robotic arm. Furthermore, we provide a comprehensive discussion on several key implemental issues. The extensive evaluation demonstrates that our learned policy outperforms the state-of-the-art methods significantly and is practically usable for real-world applications.

Planning Dense Object Packing by Pushing Objects

“…Richa et al [3] improved the computing speed of the bin-packing problem by using deep reinforcement learning. Zhao et al [2] used deep learning to place the object in a position where the next object is easy to place. Hauser et al [4] achieved a robotic packing method that considers the stability of each object based on the BL method.…”

“…Object placement planning is known to be the knapsack problem of NP-hard. Approximate methods [1] and deeplearning-based method [2] have been proposed. Nevertheless, even if the object placement problem can be solved, it is not easy to realize the planned object arrangement by a robot, especially when a dense object arrangement is planned.…”

“…Our method can realize the dense object placement with no gap between objects. Instead of directly placing objects in the *This work was not supported by any organization 1 Graduate School of Engineering Science, Osaka University, Japan, iwao@hlab.sys.es.osaka-u.ac.jp 2 National Institute of Advanced Industrial Science and Technology (AIST), Japan, harada@hlab.sys.es.osaka-u.ac.jp target place, our method first places the grasped object at a certain distance from other objects which have already been placed in a box. Then the robot pushes the target object to the side of another object to realize the object placement without gaps as shown in Fig.…”

“…(1) directly placing the object in the target position, (2) placing the object at a certain distance from other objects and then pushing it to realize the object placement without gap, and (3) placing multiple objects at a certain distance from others and then simultaneously pushing multiple objects to realize the dense object placement. A set of actions where the robot realizes the object placement which consumes the minimum area on the bottom surface of the container with minimum number of action sequences without causing the collision with the container wall is selected.…”