A Sim-to-Real Pipeline for Deep Reinforcement Learning for Autonomous Robot Navigation in Cluttered Rough Terrain

Han, Hu; Zhang, Kaicheng; Tan, Aaron Hao; Ruan, Michael; Agia, Christopher; Nejat, Goldie

doi:10.1109/lra.2021.3093551

Cited by 57 publications

(20 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The full name of sim-to-real is simulation to reality, which is a branch of reinforcement learning and a kind of transfer learning [52]. In the field of robotics or autonomous driving, the main problem that transfer learning solves is that of how to directly allow the autonomous systems or agents to interact with the virtual environment and the real environment [53,54]. Reinforcement learning is considered as a promising direction for driving policy learning.…”

Section: Discussion Of Limitations and Future Workmentioning

confidence: 99%

Self-Optimizing Path Tracking Controller for Intelligent Vehicles Based on Reinforcement Learning

Xie

Song

et al. 2021

Symmetry

View full text Add to dashboard Cite

The path tracking control system is a crucial component for autonomous vehicles; it is challenging to realize accurate tracking control when approaching a wide range of uncertain situations and dynamic environments, particularly when such control must perform as well as, or better than, human drivers. While many methods provide state-of-the-art tracking performance, they tend to emphasize constant PID control parameters, calibrated by human experience, to improve tracking accuracy. A detailed analysis shows that PID controllers inefficiently reduce the lateral error under various conditions, such as complex trajectories and variable speed. In addition, intelligent driving vehicles are highly non-linear objects, and high-fidelity models are unavailable in most autonomous systems. As for the model-based controller (MPC or LQR), the complex modeling process may increase the computational burden. With that in mind, a self-optimizing, path tracking controller structure, based on reinforcement learning, is proposed. For the lateral control of the vehicle, a steering method based on the fusion of the reinforcement learning and traditional PID controllers is designed to adapt to various tracking scenarios. According to the pre-defined path geometry and the real-time status of the vehicle, the interactive learning mechanism, based on an RL framework (actor–critic—a symmetric network structure), can realize the online optimization of PID control parameters in order to better deal with the tracking error under complex trajectories and dynamic changes of vehicle model parameters. The adaptive performance of velocity changes was also considered in the tracking process. The proposed controlling approach was tested in different path tracking scenarios, both the driving simulator platforms and on-site vehicle experiments have verified the effects of our proposed self-optimizing controller. The results show that the approach can adaptively change the weights of PID to maintain a tracking error (simulation: within ±0.071 m; realistic vehicle: within ±0.272 m) and steering wheel vibration standard deviations (simulation: within ±0.04°; realistic vehicle: within ±80.69°); additionally, it can adapt to high-speed simulation scenarios (the maximum speed is above 100 km/h and the average speed through curves is 63–76 km/h).

show abstract

Section: Discussion Of Limitations and Future Workmentioning

confidence: 99%

Self-Optimizing Path Tracking Controller for Intelligent Vehicles Based on Reinforcement Learning

Xie

Song

et al. 2021

Symmetry

View full text Add to dashboard Cite

show abstract

“…For the test dataset, a stride of 25 samples is chosen to avoid overlap. In the test dataset, the time duration of the first trajectory is 23 s, and the second trajectory duration is 30 s, resulting in dimensions of [23,6,25] and [30,6,25].…”

Section: Data Collection and Preprocessingmentioning

confidence: 99%

“…In other related navigation domains, machine learning (ML) and deep learning (DL) algorithms are used to improve the overall navigation performance. In [23][24][25], a deep learning approach is used for robot indoor navigation. Human activity recognition [26][27][28] and smartphone location recognition (SLR) [29] algorithms based on ML/DL were shown to improve the accuracy of PDR by using it as a prior [30,31].…”

Section: Introductionmentioning

confidence: 99%

QuadNet: A Hybrid Framework for Quadrotor Dead Reckoning

Shurin

Klein

2022

Sensors

View full text Add to dashboard Cite

Quadrotor usage is continuously increasing for both civilian and military applications such as surveillance, mapping, and deliveries. Commonly, quadrotors use an inertial navigation system combined with a global navigation satellite systems receiver for outdoor applications and a camera for indoor/outdoor applications. For various reasons, such as lighting conditions or satellite signal blocking, the quadrotor’s navigation solution depends only on the inertial navigation system solution. As a consequence, the navigation solution drifts in time due to errors and noises in the inertial sensor measurements. To handle such situations and bind the solution drift, the quadrotor dead reckoning (QDR) approach utilizes pedestrian dead reckoning principles. To that end, instead of flying the quadrotor in a straight line trajectory, it is flown in a periodic motion, in the vertical plane, to enable peak-to-peak (two local maximum points within the cycle) distance estimation. Although QDR manages to improve the pure inertial navigation solution, it has several shortcomings as it requires calibration before usage, provides only peak-to-peak distance, and does not provide the altitude of the quadrotor. To circumvent these issues, we propose QuadNet, a hybrid framework for quadrotor dead reckoning to estimate the quadrotor’s three-dimensional position vector at any user-defined time rate. As a hybrid approach, QuadNet uses both neural networks and model-based equations during its operation. QuadNet requires only the inertial sensor readings to provide the position vector. Experimental results with DJI’s Matrice 300 quadrotor are provided to show the benefits of using the proposed approach.

show abstract

“…In our work, we focus on the effectiveness of the policy switching criterion such that the overall sample efficiency and final performances can be both preserved. In addition to offline RL, there are also some other works that aim to reduce the interaction frequency with the environment rather than the switching cost [7,10], which is parallel to our focus.…”

Section: Related Workmentioning

confidence: 99%

A Benchmark for Low-Switching-Cost Reinforcement Learning

Xu¹,

Yancheng²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

A ubiquitous requirement in many practical reinforcement learning (RL) applications, including medical treatment, recommendation system, education and robotics, is that the deployed policy that actually interacts with the environment cannot change frequently. Such an RL setting is called low-switching-cost RL, i.e., achieving the highest reward while reducing the number of policy switches during training. Despite the recent trend of theoretical studies aiming to design provably efficient RL algorithms with low switching costs, none of the existing approaches have been thoroughly evaluated in popular RL testbeds. In this paper, we systematically studied a wide collection of policy-switching approaches, including theoretically guided criteria, policy-difference-based methods, and non-adaptive baselines. Through extensive experiments on a medical treatment environment, the Atari games, and robotic control tasks, we present the first empirical benchmark for low-switchingcost RL and report novel findings on how to decrease the switching cost while maintain a similar sample efficiency to the case without the low-switching-cost constraint. We hope this benchmark could serve as a starting point for developing more practically effective low-switching-cost RL algorithms. We release our code and complete results in https:// sites.google.com/ view/ low-switching-cost-rl.

show abstract

A Sim-to-Real Pipeline for Deep Reinforcement Learning for Autonomous Robot Navigation in Cluttered Rough Terrain

Cited by 57 publications

References 25 publications

Self-Optimizing Path Tracking Controller for Intelligent Vehicles Based on Reinforcement Learning

Self-Optimizing Path Tracking Controller for Intelligent Vehicles Based on Reinforcement Learning

QuadNet: A Hybrid Framework for Quadrotor Dead Reckoning

A Benchmark for Low-Switching-Cost Reinforcement Learning

Contact Info

Product

Resources

About