Specifying Dual-Arm Robot Planning Problems Through Natural Language and Demonstration

Behrens, Jan Kristof; Štěpánová, Karla; Lange, Ralph; Škoviera, Radoslav

doi:10.1109/lra.2019.2898714

Cited by 13 publications

(2 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In other works, such as [9], gestures are used to point to objects that should be manipulated. In general, gestures are used in current works as one-word commands and not as real sentences containing verbs, subjects and further parameters.…”

Section: Related Workmentioning

confidence: 99%

Communicating human intent to a robotic companion by multi-type gesture sentences

Petr¹,

Behrens²,

Štěpánová³

et al. 2023

Preprint

View full text Add to dashboard Cite

Human-Robot collaboration in home and industrial workspaces is on the rise. However, the communication between robots and humans is a bottleneck. Although people use a combination of different types of gestures to complement speech, only a few robotic systems utilize gestures for communication. In this paper, we propose a gesture pseudolanguage and show how multiple types of gestures can be combined to express human intent to a robot (i.e., expressing both the desired action and its parameters -e.g., pointing to an object and showing that the object should be emptied into a bowl). The demonstrated gestures and the perceived tabletop scene (object poses detected by CosyPose) are processed in real-time) to extract the human's intent. We utilize behavior trees to generate reactive robot behavior that handles various possible states of the world (e.g., a drawer has to be opened before an object is placed into it) and recovers from errors (e.g., when the scene changes). Furthermore, our system enables switching between direct teleoperation of the end-effector and high-level operation using the proposed gesture sentences. The system is evaluated on increasingly complex tasks using a real 7-DoF Franka Emika Panda manipulator. Controlling the robot via action gestures lowered the execution time by up to 60%, compared to direct teleoperation.

show abstract

Section: Related Workmentioning

confidence: 99%

Communicating human intent to a robotic companion by multi-type gesture sentences

Petr¹,

Behrens²,

Štěpánová³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Q3: Instead of directly specifying a sequence of production routines into a planning domain languages (e.g., a constraint problem), a multi-model learning method can be used for robot programming. In particular, a combination of learning from demonstration and requirements specification through natural language has been shown to be effective in preparing robot assembly planning domains for flexible manufacturing ( Behrens et al, 2019b ).…”

Section: Applicationsmentioning

confidence: 99%

Combining Task and Motion Planning: Challenges and Guidelines

2021

View full text Add to dashboard Cite

Combined Task and Motion Planning (TAMP) is an area where no one-fits-all solution can exist. Many aspects of the domain, as well as operational requirements, have an effect on how algorithms and representations are designed. Frequently, trade-offs have to be made to build a system that is effective. We propose five research questions that we believe need to be answered to solve real-world problems that involve combined TAMP. We show which decisions and trade-offs should be made with respect to these research questions, and illustrate these on examples of existing application domains. By doing so, this article aims to provide a guideline for designing combined TAMP solutions that are adequate and effective in the target scenario.

show abstract

A Multi-modal Framework for Robots to Learn Manipulation Tasks from Human Demonstrations

Yin

Zhang

2023

J Intell Robot Syst

View full text Add to dashboard Cite

Specifying Dual-Arm Robot Planning Problems Through Natural Language and Demonstration

Cited by 13 publications

References 12 publications

Communicating human intent to a robotic companion by multi-type gesture sentences

Communicating human intent to a robotic companion by multi-type gesture sentences

Combining Task and Motion Planning: Challenges and Guidelines

A Multi-modal Framework for Robots to Learn Manipulation Tasks from Human Demonstrations

Contact Info

Product

Resources

About