Adaptive Variance for Changing Sparse-Reward Environments

Lin, Xingyu; Guo, Pengsheng; Florensa, Carlos; Held, David

doi:10.1109/icra.2019.8793650

Cited by 4 publications

(12 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Robots are now being utilized in outer space missions, medical surgery, meal delivery in hospitals [49], and so on. However, often robots need to adapt to non-stationary operating conditions-for example, a ground robot/rover must adapt its walking gait to changing terrain conditions [44] or friction coefficients of surface [41].…”

Section: Roboticsmentioning

confidence: 99%

“…Robotic environments characterized by changing conditions and sparse rewards are particularly hard to learn, because, often, the reinforcement to the RL agent is a small value and is also obtained at the end of the task. Reference [41] focuses on learning in robotic arms where object manipulation is characterized by sparse-reward environments. The robotic arm is tasked with moving or manipulating objects that are placed at fixed positions on a table.…”

Section: Roboticsmentioning

confidence: 99%

“…In these tasks, often, dynamic adaptation to the surface friction and changed placement of objects on the table is tough. Reference [41] adapts the TRPO algorithm for dealing with these changing operating conditions. The robotic arm RL agent is modeled as a continuous state-action space MDP.…”

Section: Roboticsmentioning

confidence: 99%

“…In a continuous stateaction space setting, the policy is parameterized by Gaussian distribution. Reference [41] proposes a strategy to adjust the variance of this Gaussian policy to adapt to environment changes.…”

Section: Roboticsmentioning

confidence: 99%

“…It is tough to extend the above works to more general settings. Some works that are generalizable are References [3,4,41,52,70]. All these works are summarized in Table 3.…”

Section: Remarksmentioning

confidence: 99%

See 4 more Smart Citations

A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environments

Padakandla

2021

ACM Comput. Surv.

View full text Add to dashboard Cite

Reinforcement learning (RL) algorithms find applications in inventory control, recommender systems, vehicular traffic management, cloud computing, and robotics. The real-world complications arising in these domains makes them difficult to solve with the basic assumptions underlying classical RL algorithms. RL agents in these applications often need to react and adapt to changing operating conditions. A significant part of research on single-agent RL techniques focuses on developing algorithms when the underlying assumption of stationary environment model is relaxed. This article provides a survey of RL methods developed for handling dynamically varying environment models. The goal of methods not limited by the stationarity assumption is to help autonomous agents adapt to varying operating conditions. This is possible either by minimizing the rewards lost during learning by RL agent or by finding a suitable policy for the RL agent that leads to efficient operation of the underlying system. A representative collection of these algorithms is discussed in detail in this work along with their categorization and their relative merits and demerits. Additionally, we also review works that are tailored to application domains. Finally, we discuss future enhancements for this field.

show abstract

Section: Roboticsmentioning

confidence: 99%

Section: Roboticsmentioning

confidence: 99%

Section: Roboticsmentioning

confidence: 99%

Section: Roboticsmentioning

confidence: 99%

“…It is tough to extend the above works to more general settings. Some works that are generalizable are References [3,4,41,52,70]. All these works are summarized in Table 3.…”

Section: Remarksmentioning

confidence: 99%

See 3 more Smart Citations

A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environments

Padakandla

2021

ACM Comput. Surv.

View full text Add to dashboard Cite

show abstract

Vision-Based Goal-Conditioned Policies for Underwater Navigation in the Presence of Obstacles

Manderson¹,

Higuera²,

Wapnick³

et al. 2020

Robotics: Science and Systems XVI

View full text Add to dashboard Cite

We present Nav2Goal, a data-efficient and end-toend learning method for goal-conditioned visual navigation. Our technique is used to train a navigation policy that enables a robot to navigate close to sparse geographic waypoints provided by a user without any prior map, all while avoiding obstacles and choosing paths that cover user-informed regions of interest. Our approach is based on recent advances in conditional imitation learning. General-purpose safe and informative actions are demonstrated by a human expert. The learned policy is subsequently extended to be goal-conditioned by training with hindsight relabelling, guided by the robot's relative localization system, which requires no additional manual annotation. We deployed our method on an underwater vehicle in the open ocean to collect scientifically relevant data of coral reefs, which allowed our robot to operate safely and autonomously, even at very close proximity to the coral. Our field deployments have demonstrated over a kilometer of autonomous visual navigation, where the robot reaches on the order of 40 waypoints, while collecting scientifically relevant data. This is done while travelling within 0.5 m altitude from sensitive corals and exhibiting significant learned agility to overcome turbulent ocean conditions and to actively avoid collisions.

show abstract

Goal-conditioned Imitation Learning

Ding¹,

Florensa²,

Abbeel³

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Designing rewards for Reinforcement Learning (RL) is challenging because it needs to convey the desired task, be efficient to optimize, and be easy to compute. The latter is particularly problematic when applying RL to robotics, where detecting whether the desired configuration is reached might require considerable supervision and instrumentation. Furthermore, we are often interested in being able to reach a wide range of configurations, hence setting up a different reward every time might be unpractical. Methods like Hindsight Experience Replay (HER) have recently shown promise to learn policies able to reach many goals, without the need of a reward. Unfortunately, without tricks like resetting to points along the trajectory, HER might require many samples to discover how to reach certain areas of the state-space. In this work we investigate different approaches to incorporate demonstrations to drastically speed up the convergence to a policy able to reach any goal, also surpassing the performance of an agent trained with other Imitation Learning algorithms. Furthermore, we show our method can also be used when the available expert trajectories do not contain the actions, which can leverage kinesthetic or third person demonstration. The code is available 2 . * Equal contribution 2 https://sites.google.com/view/goalconditioned-il/ Preprint. Under review.

show abstract

Adaptive Variance for Changing Sparse-Reward Environments

Cited by 4 publications

References 28 publications

A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environments

A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environments

Vision-Based Goal-Conditioned Policies for Underwater Navigation in the Presence of Obstacles

Goal-conditioned Imitation Learning

Contact Info

Product

Resources

About