Interactive Language: Talking to Robots in Real Time

Lynch, Corey; Wahid, Ayzaan; Tompson, Jonathan; Ding, Tianli; Betker, James; Baruch, Robert; Armstrong, T. P.; Florence, Pete

doi:10.48550/arxiv.2210.06407

Cited by 8 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In more complex settings, the agent interacts with humans using natural language. In Lynch et al (2022) the authors present a RL and imitation based framework, Interactive Language, that is capable of continuously adjusting its behavior to natural language based instructions in a real-time interactive setting. There have also been a recent wave of datasets and benchmarks created by utilizing 3D household simulators and crowd sourcing tools to collect large-scale task-oriented dialogue aimed at improving the interactive language capabilities of embodied task-oriented agents Padmakumar et al (2022), Gao et al (2022), Team et al (2021).…”

Section: Rl For Communication In Task-oriented Embodied Agentsmentioning

confidence: 99%

Learning to generate pointing gestures in situated embodied conversational agents

et al. 2023

View full text Add to dashboard Cite

One of the main goals of robotics and intelligent agent research is to enable them to communicate with humans in physically situated settings. Human communication consists of both verbal and non-verbal modes. Recent studies in enabling communication for intelligent agents have focused on verbal modes, i.e., language and speech. However, in a situated setting the non-verbal mode is crucial for an agent to adapt flexible communication strategies. In this work, we focus on learning to generate non-verbal communicative expressions in situated embodied interactive agents. Specifically, we show that an agent can learn pointing gestures in a physically simulated environment through a combination of imitation and reinforcement learning that achieves high motion naturalness and high referential accuracy. We compared our proposed system against several baselines in both subjective and objective evaluations. The subjective evaluation is done in a virtual reality setting where an embodied referential game is played between the user and the agent in a shared 3D space, a setup that fully assesses the communicative capabilities of the generated gestures. The evaluations show that our model achieves a higher level of referential accuracy and motion naturalness compared to a state-of-the-art supervised learning motion synthesis model, showing the promise of our proposed system that combines imitation and reinforcement learning for generating communicative gestures. Additionally, our system is robust in a physically-simulated environment thus has the potential of being applied to robots.

show abstract

Section: Rl For Communication In Task-oriented Embodied Agentsmentioning

confidence: 99%

Learning to generate pointing gestures in situated embodied conversational agents

et al. 2023

View full text Add to dashboard Cite

show abstract

“…LLMs have been showing incredible potential in areas besides robotics [1], [12], [6], [13]. Human knowledge that resides in LLMs can help a robot abstract and select only suitable features, e.g.…”

Section: B Chatting With the Environmentmentioning

confidence: 99%

Chat with the Environment: Interactive Multimodal Perception using Large Language Models

Zhao¹,

Li²,

Weber³

et al. 2023

Preprint

View full text Add to dashboard Cite

Programming robot behaviour in a complex world faces challenges on multiple levels, from dextrous low-level skills to high-level planning and reasoning. Recent pre-trained Large Language Models (LLMs) have shown remarkable reasoning ability in zero-shot robotic planning. However, it remains challenging to ground LLMs in multimodal sensory input and continuous action output, while enabling a robot to interact with its environment and acquire novel information as its policies unfold. We develop a robot interaction scenario with a partially observable state, which necessitates a robot to decide on a range of epistemic actions in order to sample sensory information among multiple modalities, before being able to execute the task correctly. An interactive perception framework is therefore proposed with an LLM as its backbone, whose ability is exploited to instruct epistemic actions and to reason over the resulting multimodal sensations (vision, sound, haptics, proprioception), as well as to plan an entire task execution based on the interactively acquired information. Our study demonstrates that LLMs can provide high-level planning and reasoning skills and control interactive robot behaviour in a multimodal environment, while multimodal modules with the context of the environmental state help ground the LLMs and extend their processing ability.

show abstract

“…This has been extended to use a map and object-centric representation of the world [14]. A set of other works have used language to build various complex structures [15], [7], [16], [17].…”

Section: Related Workmentioning

confidence: 99%

StructDiffusion: Object-Centric Diffusion for Semantic Rearrangement of Novel Objects

Liu¹,

Du²,

Hermans³

et al. 2022

Preprint

View full text Add to dashboard Cite

Interactive Language: Talking to Robots in Real Time

Cited by 8 publications

References 0 publications

Learning to generate pointing gestures in situated embodied conversational agents

Learning to generate pointing gestures in situated embodied conversational agents

Chat with the Environment: Interactive Multimodal Perception using Large Language Models

StructDiffusion: Object-Centric Diffusion for Semantic Rearrangement of Novel Objects

Contact Info

Product

Resources

About