Automated Excavator Based on Reinforcement Learning and Multibody System Dynamics

Abstract

Fully autonomous earth-moving heavy equipment able to operate without human intervention can be seen as the primary goal of automated earth construction. To achieve this objective requires that the machines have the ability to adapt autonomously to complex and changing environments. Recent developments in automation have focused on the application of different machine learning approaches, of which the use of reinforcement learning algorithms is considered the most promising. The key advantage of reinforcement learning is the ability of the system to learn, adapt and work independently in a dynamic environment. This paper investigates an application of reinforcement learning algorithm for heavy mining machinery automation. To this end, the training associated with reinforcement learning is done using the multibody approach. The procedure used combines a multibody approach and proximal policy optimization with a covariance matrix adaptation learning algorithm to simulate an autonomous excavator. The multibody model includes a representation of the hydraulic system, multiple sensors observing the state of the excavator and deformable ground. The task of loading a hopper with soil taken from a chosen point on the ground is simulated. The excavator is trained to load the hopper effectively within a given time while avoiding collisions with the ground and the hopper. The proposed system demonstrates the desired behavior after short training times.

Deep Reinforcement Learning With Adversarial Training for Automated Excavation Using Depth Images

“…This approach is especially attractive for tasks in which simulations are available, because the optimal policy can be obtained from virtual samples. Recent studies have applied deep RL to excavation tasks [14], [20]. A simulator for a bucket-leveling task was developed in a previous study [20], and the efficacy of deep RL methods was investigated.…”

“…A simulator for a bucket-leveling task was developed in a previous study [20], and the efficacy of deep RL methods was investigated. In a previous study by Kurinov et al [14], a 3D simulation of the excavation task was developed, and deep RL was applied to automate the excavation task. Although they achieved promising results, the obtained policy is based on a low-dimensional state vector, which is difficult to obtain in real-world systems.…”

“…Therefore, RL removes the needs for manual design of policies or an expert-demonstration dataset. Recently, Kurinov et al investigated a framework for automating excavation based on deep RL [14] and demonstrated that they had successfully obtained…”

Data-Driven Reinforcement-Learning-Based Automatic Bucket-Filling for Wheel Loaders

“…Hodel et al [20] applied RL-based simulation methods to control the excavator to perform the bucket-leveling task. Kurinov et al [21] investigated the application of an RL algorithm for excavator automation. In the proposed system, the agent of the excavator can learn a policy by interacting with the simulated model.…”

An Extension of BIM Using AI: A Multi Working-Machines Pathfinding Solution

“…Also, Tumer describes the future construction site utilizing industry 4.0 [33]. Without a doubt, a future working site should be fully benefited from AI [34]- [36], automation technology [37], Simultaneous Localization and Mapping (SLAM) [38], and IoT [10], [11], [39], [40]. Therefore, the uncertainty degree of construction sites is reducing, and thus we consider the construction sites as known in our research.…”