Wheel Loader Scooping Controller Using Deep Reinforcement Learning

Abstract

This article presents a deep reinforcement learning-based controller for an unmanned ground vehicle with a custom-built scooping mechanism. The robot's aim is to autonomously perform earth scooping cycles with three degrees of freedom: lift, tilt and the robot's velocity. While the majority of previous studies on automated scooping processes are based on data recorded by expert operators, we present a method to autonomously control a wheel loader to perform the scooping cycle using deep reinforcement learning methods without any user-provided demonstrations. The controller's learning approach is based on the actorcritic, Deep Deterministic Policy Gradient algorithm which we use to map online sensor data as input to continuously update the actuator commands. The training of the scooping policy network is done solely in a simplified simulation environment using a virtual physics engine, which converges to an average of a 65% fill factor from the full bucket capacity and a 5 [ ] average cycle time. We illustrate the performance of the trained policy in simulations and in real-world experiments with 3 different inclination angles of the earth. An additional scooping experiment compared the performance of our controller to remote manual human control. Overall, the deep reinforcement learning-based controller exhibited good performance in terms of both achieved visually bucket fill with varying scooped earth weights of 4.1 − 7.2[ ], and a 5.1 − 7.1[ ] cycle time. The experimental results confirm the ability of our planner to fill bucket as required, indicating that our controller can be used for excavation purposes.

Continuous Control of an Underground Loader Using Deep Reinforcement Learning

“…In future field tests, the amount of spillage on the ground and the slipping of the wheels should also be measured and taken into account when comparing the overall performance. Comparison with [21] and [18] is interesting but difficult because of the large differences in the vehicles' size and strength and the materials' properties. The reinforcement learning controller in [21] achieved a fill factor of 65%, and the energy consumption was not measured.…”

“…Comparison with [21] and [18] is interesting but difficult because of the large differences in the vehicles' size and strength and the materials' properties. The reinforcement learning controller in [21] achieved a fill factor of 65%, and the energy consumption was not measured. The neural network controller in [18], which was trained by learning from demonstration, reached 81% of the filling of the bucket relative to manual loading, but neither loading time nor work was reported.…”

“…With the adaptation algorithm in [20], the network adapts from loading mediumcoarse gravel to cobble gravel, with a five-to ten-percent increase in bucket filling after 40 loadings. The first use of reinforcement learning to control a scooping mechanism was recently published [21]. Using the actor-critic algorithm and the deep deterministic policy gradient algorithm, an agent was trained to control a three-degree-of-freedom mechanism in order to fill a bucket.…”

“…Using the actor-critic algorithm and the deep deterministic policy gradient algorithm, an agent was trained to control a three-degree-of-freedom mechanism in order to fill a bucket. The authors of [21] did not consider the use of high-dimensional observation data in order to adapt to variable pile shapes or to steer the vehicle. Reinforcement learning agents are often trained in simulated environments, as they are an economical and safe way to produce large amounts of labeled data [22] before transferring to real environments.…”

Data-Driven Reinforcement-Learning-Based Automatic Bucket-Filling for Wheel Loaders

“…At each time-step, the agent takes action a according to the current environmental state S t and the policy π which is a mapping from perceived states to actions. Therefore, as a consequence of action, the environmental state transits from S t to S t+1 and the agent gets a reward r. The agent and environment generate the trajectories (S 1 ; A 1 ; R 1 ), (S 2 ; A 2 ; R 2 ), ..., (S T ; A T ; R T ) [23], until an episode is over. The basic architecture of RL is shown in Figure 8.…”