Model primitives for hierarchical lifelong reinforcement learning

Wu, Bohan; Gupta, Jayesh K.; Kochenderfer, Mykel J.

doi:10.1007/s10458-020-09451-0

Cited by 23 publications

(9 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Modularity Modular approaches in HRL have been relevantly applied to decompose a possibly complex task domain into different regions of specialization, but also to multitask DRL, in order to reuse previously learned skills across tasks [107][108][109]. Tasks may be annotated by handcrafted instructions (as "policy sketches" in [107]), and symbolic subtasks may be associated with subpolicies which a full task-specific policy aims to successfully combine.…”

Section: Hierarchical Approachesmentioning

confidence: 99%

“…Distinctively, [108] proposes to train-or provide as a prior-a stochastic temporal grammar (STG), in order to capture temporal transitions between the tasks; a STG encodes priorities of sub-tasks over others and enables to better learn how to switch between base or augmented subpolicies. The work in [109] seeks to learn a mixture of subpolicies by modeling the environment assuming given a set of (imperfect) models specialized by regions, referred to as model primitives. Each policy is specialized on those regions and the weights in the mixture correspond to the posterior probability of a model given the current state.…”

Section: Hierarchical Approachesmentioning

confidence: 99%

See 1 more Smart Citation

A Survey on Interpretable Reinforcement Learning

Glanois¹,

Weng²,

Zimmer³

et al. 2021

Preprint

View full text Add to dashboard Cite

Although deep reinforcement learning has become a promising machine learning approach for sequential decision-making problems, it is still not mature enough for high-stake domains such as autonomous driving or medical applications. In such contexts, a learned policy needs for instance to be interpretable, so that it can be inspected before any deployment (e.g., for safety and verifiability reasons). This survey provides an overview of various approaches to achieve higher interpretability in reinforcement learning (RL). To that aim, we distinguish interpretability (as a property of a model) and explainability (as a post-hoc operation, with the intervention of a proxy) and discuss them in the context of RL with an emphasis on the former notion. In particular, we argue that interpretable RL may embrace different facets: interpretable inputs, interpretable (transition/reward) models, and interpretable decision-making. Based on this scheme, we summarize and analyze recent work related to interpretable RL with an emphasis on papers published in the past 10 years. We also discuss briefly some related research areas and point to some potential promising research directions.

show abstract

Section: Hierarchical Approachesmentioning

confidence: 99%

Section: Hierarchical Approachesmentioning

confidence: 99%

A Survey on Interpretable Reinforcement Learning

Glanois¹,

Weng²,

Zimmer³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…goal space, while making RL in the lower level easier with an explicit and short-horizon goal. Recent works extended hierarchical RL to solve complex tasks [42,43,44,45,46]. Le et al [47] proposed a variant of hierarchical RL, which employs IL to learn the high-level policy to leverage expert feedback to explore the goal space more efficiently.…”

Section: Related Workmentioning

confidence: 99%

Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving

Cao

Bıyık

Wang

et al. 2020

Robotics: Science and Systems XVI

View full text Add to dashboard Cite

Autonomous driving has achieved significant progress in recent years, but autonomous cars are still unable to tackle high-risk situations where a potential accident is likely. In such near-accident scenarios, even a minor change in the vehicle's actions may result in drastically different consequences. To avoid unsafe actions in near-accident scenarios, we need to fully explore the environment. However, reinforcement learning (RL) and imitation learning (IL), two widely-used policy learning methods, cannot model rapid phase transitions and are not scalable to fully cover all the states. To address driving in near-accident scenarios, we propose a hierarchical reinforcement and imitation learning (H-REIL) approach that consists of low-level policies learned by IL for discrete driving modes, and a high-level policy learned by RL that switches between different driving modes. Our approach exploits the advantages of both IL and RL by integrating them into a unified learning framework. Experimental results and user studies suggest our approach can achieve higher efficiency and safety compared to other methods. Analyses of the policies demonstrate our high-level policy appropriately switches between different low-level policies in near-accident driving situations.

show abstract

“…The L3 level learning brings the idea of motion primitives from robotics towards generalized tracking behavior. There are many known approaches to primitive-based tracking control, both older and more recent [ 48 , 49 , 50 , 51 , 52 ]. Their taxonomy is not studied here because it has been reviewed elsewhere, e.g., in [ 1 , 2 , 3 ].…”

Section: Introductionmentioning

confidence: 99%

Trajectory Tracking within a Hierarchical Primitive-Based Learning Approach

Rădac

2022

Entropy

View full text Add to dashboard Cite

A hierarchical learning control framework (HLF) has been validated on two affordable control laboratories: an active temperature control system (ATCS) and an electrical rheostatic braking system (EBS). The proposed HLF is data-driven and model-free, while being applicable on general control tracking tasks which are omnipresent. At the lowermost level, L1, virtual state-feedback control is learned from input–output data, using a recently proposed virtual state-feedback reference tuning (VSFRT) principle. L1 ensures a linear reference model tracking (or matching) and thus, indirect closed-loop control system (CLCS) linearization. On top of L1, an experiment-driven model-free iterative learning control (EDMFILC) is then applied for learning reference input–controlled outputs pairs, coined as primitives. The primitives’ signals at the L2 level encode the CLCS dynamics, which are not explicitly used in the learning phase. Data reusability is applied to derive monotonic and safely guaranteed learning convergence. The learning primitives in the L2 level are finally used in the uppermost and final L3 level, where a decomposition/recomposition operation enables prediction of the optimal reference input assuring optimal tracking of a previously unseen trajectory, without relearning by repetitions, as it was in level L2. Hence, the HLF enables control systems to generalize their tracking behavior to new scenarios by extrapolating their current knowledge base. The proposed HLF framework endows the CLCSs with learning, memorization and generalization features which are specific to intelligent organisms. This may be considered as an advancement towards intelligent, generalizable and adaptive control systems.

show abstract

Model primitives for hierarchical lifelong reinforcement learning

Cited by 23 publications

References 34 publications

A Survey on Interpretable Reinforcement Learning

A Survey on Interpretable Reinforcement Learning

Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving

Trajectory Tracking within a Hierarchical Primitive-Based Learning Approach

Contact Info

Product

Resources

About