Teaching a Robot Tasks of Arbitrary Complexity via Human Feedback

Wang, Guan; Trimbach, Carl; Lee, Jun Ki; Ho, Mark K.; Littman, Michael L.

doi:10.1145/3319502.3374824

Cited by 8 publications

(5 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Alternative decompositions include subtask sequences called sketches [2], which are less expressive than RMs (the subtasks are run in a single order) and are not hierarchically composable. Context-free grammars defining a subset of English [12], formal languages [26,33,53] and logic-based algebras [39] have been also used to model task composability.…”

Section: Related Workmentioning

confidence: 99%

“…The so-called 'online' method by Matiisen et al [36] also keeps an estimate of the average return of the different tasks, but it is not presented in an HRL scenario. Wang et al [53] introduce a method that initially learns linear temporal logic formulas for simple tasks, and progressively switches to harder ones leveraging previously learned formulas using a set of predefined templates. Deriving HRMs from (learned) templates is an interesting direction for future work.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Hierarchies of Reward Machines

Furelos-Blanco¹,

Law²,

Jönsson³

et al. 2022

Preprint

View full text Add to dashboard Cite

Reward machines (RMs) are a recent formalism for representing the reward function of a reinforcement learning task through a finite-state machine whose edges encode landmarks of the task using high-level events. The structure of RMs enables the decomposition of a task into simpler and independently solvable subtasks that help tackle long-horizon and/or sparse reward tasks. We propose a formalism for further abstracting the subtask structure by endowing an RM with the ability to call other RMs, thus composing a hierarchy of RMs (HRM). We exploit HRMs by treating each call to an RM as an independently solvable subtask using the options framework, and describe a curriculum-based method to induce HRMs from example traces observed by the agent. Our experiments reveal that exploiting a handcrafted HRM leads to faster convergence than with a flat HRM, and that learning an HRM is more scalable than learning an equivalent flat HRM.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Hierarchies of Reward Machines

Furelos-Blanco¹,

Law²,

Jönsson³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Other work [8], [15], [16] has also considered grounding natural language to LTL task specifications with improved accuracy using example trajectories and human-feedback, using formal verification and optimization methods. Patel et al [15] introduced a semi-supervised method for grounding language to LTL via generative models and formal verification, considering whether candidate LTL satisfies trajectories.…”

Section: Related Workmentioning

confidence: 99%

“…Patel et al [15] introduced a semi-supervised method for grounding language to LTL via generative models and formal verification, considering whether candidate LTL satisfies trajectories. Wang et al [16] introduced an algorithm for learning LTL task specifications of arbitrary complexity from human-feedback by a process of eliminating sub-optimal trajectories. Danas et al [8] contributed a three step process for recovering from language grounding errors by performing beam search within a Seq2Seq model to determine the top most likely LTL formulae, differentiating trajectories via maximal semantic differencing, and finally requesting human feedback to clarify which trajectories match the desired task specification.…”

Section: Related Workmentioning

confidence: 99%

Generalizing to New Domains by Mapping Natural Language to Lifted LTL

Hsiung¹,

Hiloni²,

Chu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent work on using natural language to specify commands to robots has grounded that language to LTL. However, mapping natural language task specifications to LTL task specifications using language models require probability distributions over finite vocabulary. Existing state-of-the-art methods have extended this finite vocabulary to include unseen terms from the input sequence to improve output generalization. However, novel out-of-vocabulary atomic propositions cannot be generated using these methods. To overcome this, we introduce an intermediate contextual query representation which can be learned from single positive task specification examples, associating a contextual query with an LTL template. We demonstrate that this intermediate representation allows for generalization over unseen object references, assuming accurate groundings are available. We compare our method of mapping natural language task specifications to intermediate contextual queries against state-of-the-art CopyNet models capable of translating natural language to LTL, by evaluating whether correct LTL for manipulation and navigation task specifications can be output, and show that our method outperforms the CopyNet model on unseen object references. We demonstrate that the grounded LTL our method outputs can be used for planning in a simulated OO-MDP environment. Finally, we discuss some common failure modes encountered when translating natural language task specifications to grounded LTL.

show abstract

“…Another line of work is to consider human prior knowledge of task decomposition to achieve a form of curriculum learning for more complex tasks (Wang et al, 2020 ). Human input to RL has also been used in combination with policy search methods and to improve robot skills on a trajectory level (Celemin and Ruiz-del Solar, 2016 , 2019 ; Celemin et al, 2019 ).…”

Section: Related Workmentioning

confidence: 99%

Multi-Channel Interactive Reinforcement Learning for Sequential Tasks

et al. 2020

View full text Add to dashboard Cite

The ability to learn new tasks by sequencing already known skills is an important requirement for future robots. Reinforcement learning is a powerful tool for this as it allows for a robot to learn and improve on how to combine skills for sequential tasks. However, in real robotic applications, the cost of sample collection and exploration prevent the application of reinforcement learning for a variety of tasks. To overcome these limitations, human input during reinforcement can be beneficial to speed up learning, guide the exploration and prevent the choice of disastrous actions. Nevertheless, there is a lack of experimental evaluations of multi-channel interactive reinforcement learning systems solving robotic tasks with input from inexperienced human users, in particular for cases where human input might be partially wrong. Therefore, in this paper, we present an approach that incorporates multiple human input channels for interactive reinforcement learning in a unified framework and evaluate it on two robotic tasks with 20 inexperienced human subjects. To enable the robot to also handle potentially incorrect human input we incorporate a novel concept for self-confidence, which allows the robot to question human input after an initial learning phase. The second robotic task is specifically designed to investigate if this self-confidence can enable the robot to achieve learning progress even if the human input is partially incorrect. Further, we evaluate how humans react to suggestions of the robot, once the robot notices human input might be wrong. Our experimental evaluations show that our approach can successfully incorporate human input to accelerate the learning process in both robotic tasks even if it is partially wrong. However, not all humans were willing to accept the robot's suggestions or its questioning of their input, particularly if they do not understand the learning process and the reasons behind the robot's suggestions. We believe that the findings from this experimental evaluation can be beneficial for the future design of algorithms and interfaces of interactive reinforcement learning systems used by inexperienced users.

show abstract

Teaching a Robot Tasks of Arbitrary Complexity via Human Feedback

Cited by 8 publications

References 12 publications

Hierarchies of Reward Machines

Hierarchies of Reward Machines

Generalizing to New Domains by Mapping Natural Language to Lifted LTL

Multi-Channel Interactive Reinforcement Learning for Sequential Tasks

Contact Info

Product

Resources

About