Data-Driven Reinforcement-Learning-Based Automatic Bucket-Filling for Wheel Loaders

Abstract

Automation of bucket-filling is of crucial significance to the fully automated systems for wheel loaders. Most previous works are based on a physical model, which cannot adapt to the changeable and complicated working environment. Thus, in this paper, a data-driven reinforcement-learning (RL)-based approach is proposed to achieve automatic bucket-filling. An automatic bucket-filling algorithm based on Q-learning is developed to enhance the adaptability of the autonomous scooping system. A nonlinear, non-parametric statistical model is also built to approximate the real working environment using the actual data obtained from tests. The statistical model is used for predicting the state of wheel loaders in the bucket-filling process. Then, the proposed algorithm is trained on the prediction model. Finally, the results of the training confirm that the proposed algorithm has good performance in adaptability, convergence, and fuel consumption in the absence of a physical model. The results also demonstrate the transfer learning capability of the proposed approach. The proposed method can be applied to different machine-pile environments.

Automating the Short-Loading Cycle: Survey and Integration Framework

“…Q-learning has also been applied to the bucket-filling problem. This was achieved by removing the physical model of the wheel loader and instead using a statistical model predicting the state of the wheel loader at some time step [34]. From this model, an agent was trained to perform the bucket-filling process.…”

“…Some previous work has explicitly leveraged FSMs within their solution; alternatively, the solution can be described as an FSM, for example, decomposing the scooping task into fuzzy behaviors using finite-state machines [22], flow charts of modeling and action [34], or step-based solution [6].…”

Autonomous detection and loading of ore piles with load–haul–dump machines in Room & Pillar mines

“…In Fernando et al (2019), the method from the previous paper is explained with more detail, while the four‐step‐state machine from Dobson et al (2017) is added to the method. In Huang et al (2021), an automatic bucket‐filling algorithm based on Q‐learning, a reinforcement learning algorithm, is proposed. The state representation used considers the velocity, the tilt cylinder pressure, and the lift cylinder pressure.…”

“…The different described works use different sensor setups. In Huang et al (2021), a global positioning system (GPS) attached to the wheel loader is used. In Hejase and Ozguner (2022) several simulated sensors are considered (GPS, stereo cameras, LIDAR, and RADAR), but only the GPS is used by the system.…”

“…A load–haul–dump (LHD) is an articulated vehicle, with a frontal bucket, used for the loading, hauling, and dumping of material inside mines. LHD automation is considered to grow in the near future (Cognitive‐Market‐Research, 2023) Both autonomous navigation of LHDs and autonomous loading have been the subject of research (Androulakis, 2021; Backman et al, 2021; Dobson et al, 2017; Gu et al, 2019; Huang et al, 2021; Li & Zhan, 2018; Marshall, Barfoot, et al, 2008; Mascaró et al, 2021; Tampier et al, 2021; Xiao et al, 2022).…”