Semantic constraints to represent common sense required in household actions for multi-modal Learning-from-observation robot

Ikeuchi, Katsushi; Wake, Naoki; Arakawa, Riku; Sasabuchi, Kazuhiro; Takamatsu, Jun

doi:10.48550/arxiv.2103.02201

Cited by 5 publications

(8 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, we designed a prompt to make GPT-4 decompose given instruction into a sequence of robot tasks [10]. The set of robot tasks was defined based on the change in the motion constraints on manipulated object [50], [51], following the Kuhn-Tucker theory [52]. This definition allows us to theoretically establish a necessary and sufficient set of robot actions for object manipulation.…”

Section: A Symbolic Task Plannermentioning

confidence: 99%

Interactive Task Encoding System for Learning-from-Observation

Wake¹,

Kanehira²,

Sasabuchi³

et al. 2023

2023 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM)

View full text Add to dashboard Cite

This technical report explores the ability of Chat-GPT in recognizing emotions from text, which can be the basis of various applications like interactive chatbots, data annotation, and mental health analysis. While prior research has shown ChatGPT's basic ability in sentiment analysis, its performance in more nuanced emotion recognition is not yet explored. Here, we conducted experiments to evaluate its performance of emotion recognition across different datasets and emotion labels. Our findings indicate a reasonable level of reproducibility in its performance, with noticeable improvement through fine-tuning. However, the performance varies with different emotion labels and datasets, highlighting an inherent instability and possible bias. The choice of dataset and emotion labels significantly impacts ChatGPT's emotion recognition performance. This paper sheds light on the importance of dataset and label selection, and the potential of fine-tuning in enhancing ChatGPT's emotion recognition capabilities, providing a groundwork for better integration of emotion analysis in applications using ChatGPT.

show abstract

Section: A Symbolic Task Plannermentioning

confidence: 99%

Interactive Task Encoding System for Learning-from-Observation

Wake¹,

Kanehira²,

Sasabuchi³

et al. 2023

2023 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM)

View full text Add to dashboard Cite

show abstract

“…The appropriate set of robot actions should depend on the application and the implementation of robotic software, and thus should be customized by experimenters. In the case of our in-house learning-from-observation application [30], [31], we define robot actions as functions that change the motion constraints on the objects being manipulated, based on the Kuhn-Tucker theory [32]. This application enables us to prepare a necessary and sufficient set of robot actions for object manipulation in theory.…”

Section: B the Definition Of Robot Actionsmentioning

confidence: 99%

A Learning-from-Observation Framework: One-Shot Robot Teaching for Grasp-Manipulation-Release Household Operations

Wake

Arakawa

Yanokura

et al. 2021

2021 IEEE/SICE International Symposium on System Integration (SII)

Self Cite

View full text Add to dashboard Cite

This paper aims to provide a specific example of how OpenAI's ChatGPT can be used in a few-shot setting to convert natural language instructions into a sequence of executable robot actions (Fig. 1). Generating programs for robots from natural language instructions is an attractive goal, but the practical application using ChatGPT is still in its early stages, and there is no established methodology yet. Here, we have designed easy-to-customize input prompts for ChatGPT that meet common requirements in many practical applications, including: 1) easy integration with robot execution systems or visual recognition programs, 2) applicability to various environments, and 3) the ability to provide long-step instructions while minimizing the impact of ChatGPT's token limit. Specifically, the prompts encourage ChatGPT to 1) output a sequence of predefined robot actions with explanations in a readable JSON format, 2) represent the operating environment in a formalized style, and 3) infer and output the updated state of the operating environment as the result of each operation, which will be input with the next instruction to allow ChatGPT to work based solely on the memory of the latest operations. Through experiments, we confirmed that the proposed prompts allow ChatGPT to act in accordance with the requirements in various environments. Additionally, we observed that ChatGPT's conversational ability allows users to adjust its output with natural language feedback, which is crucial for developing an application that is both safe and robust while providing a user-friendly interface. Users can easily customize the prompts as templates. The contribution of this paper is to provide and publish the prompts, which are generic enough to be easily modified to fit the requirements of each experimenter, thereby providing practical knowledge to the robotics research community. Our prompts and source code for using them are open-source and publicly available at https://github.com/microsoft/ChatGPT-Robot-Manipulation-Prompts. Fig. 1. This paper shows practical prompts for ChatGPT to generate for translating a sequences of executable robot actions from multi-step human instructions in various environments.

show abstract

“…Manipulations can be classified based on a physical constraint. In the previous study [8], a manipulation group is defined to have a common admissible/inadmissible direction, along which the object can/cannot move. For example, several manipulations, such as drawer opening, plate sliding, and pole pulling, belong to the same group because the object's admissible motion directions are constrained under a linear guide.…”

Section: Introductionmentioning

confidence: 99%

“…Toward a robot system capable of performing a wide range of manipulations, it is important to design the generalized policy for each manipulation group. Given that household manipulations can be classified based on their common constraints [8], the key to the generalized policies is to design an environment and reward focusing on a common characteristic within each constraint group. This study validated the concept of the constraint-aware policy for two fundamental physical constraints, those being a prismatic and revolute joint.…”

mentioning

confidence: 99%

Constraint-Aware Policy for Compliant Manipulation

Saito,

Sasabuchi,

Wake

et al. 2023

Robotics

View full text Add to dashboard Cite

Robot manipulation in a physically constrained environment requires compliant manipulation. Compliant manipulation is a manipulation skill to adjust hand motion based on the force imposed by the environment. Recently, reinforcement learning (RL) has been applied to solve household operations involving compliant manipulation. However, previous RL methods have primarily focused on designing a policy for a specific operation that limits their applicability and requires separate training for every new operation. We propose a constraint-aware policy that is applicable to various unseen manipulations by grouping several manipulations together based on the type of physical constraint involved. The type of physical constraint determines the characteristic of the imposed force direction; thus, a generalized policy is trained in the environment and reward designed on the basis of this characteristic. This paper focuses on two types of physical constraints: prismatic and revolute joints. Experiments demonstrated that the same policy could successfully execute various compliant manipulation operations, both in the simulation and reality. We believe this study is the first step toward realizing a generalized household robot.

show abstract

Semantic constraints to represent common sense required in household actions for multi-modal Learning-from-observation robot

Cited by 5 publications

References 31 publications

Interactive Task Encoding System for Learning-from-Observation

Interactive Task Encoding System for Learning-from-Observation

A Learning-from-Observation Framework: One-Shot Robot Teaching for Grasp-Manipulation-Release Household Operations

Constraint-Aware Policy for Compliant Manipulation

Contact Info

Product

Resources

About