Approximate Task Tree Retrieval in a Knowledge Network for Robotic Cooking

Sakib, Md. Sadman; Paulius, David; Sun, Yu

doi:10.1109/lra.2022.3191068

Cited by 11 publications

(4 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Language translation in the context of LLMs and planning involves transforming natural language instructions into structured planning languages (Wong et al 2023;Kelly et al 2023;Pan et al 2023;Xie et al 2023;Yang, Ishay, and Lee 2023;Lin et al 2023c;Sakib and Sun 2023;Yang et al 2023b;Parakh et al 2023;Yang et al 2023a;Dai et al 2023;Ding et al 2023b;Zelikman et al 2023;Xu et al 2023b;Chen et al 2023a;You et al 2023) such as PDDL, and vice versa, utilizing in-context learning techniques (Guan et al 2023). This capability effectively bridges the gap between human linguistic expression and machine-understandable formats, enhancing intuitive and efficient planning processes.…”

Section: Language Translationmentioning

confidence: 99%

“…Efforts to generate multimodal, text, and image-based goalconditioned plans are exemplified by (Lu et al 2023b). Additionally, a subset of studies in this survey investigates the fine-tuning of seq2seq, code-based language models (Pallagani et al 2022(Pallagani et al , 2023b, which are noted for their advanced Application of LLMs in Planning Language Translation ( 23) Xie et al 2023;Guan et al 2023;Chalvatzaki et al 2023;Yang, Ishay, and Lee 2023;Wong et al 2023;Kelly et al 2023;Lin et al 2023c;Sakib and Sun 2023;Yang et al 2023b;Parakh et al 2023;Dai et al 2023;Yang et al 2023a;Shirai et al 2023;Ding et al 2023b;Zelikman et al 2023;Pan et al 2023;Xu et al 2023b;Brohan et al 2023;Yang, Gaglione, and Topcu 2022;Chen et al 2023a;You et al 2023) Plan Generation (53) (Sermanet et al 2023;Li et al 2023b;Pallagani et al 2022;Silver et al 2023;Pallagani et al 2023b;Arora and Kambhampati 2023;Fabiano et al 2023;Chalvatzaki et al 2023;Gu et al 2023;Silver et al 2022;Hao et al 2023a;Lin et al 2023b;Yuan et al 2023b;Gandhi, Sadigh, and Goodman 2023;…”

Section: Plan Generationmentioning

confidence: 99%

See 1 more Smart Citation

On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)

Pallagani,

Muppasani,

Roy

et al. 2024

ICAPS

View full text Add to dashboard Cite

Automated Planning and Scheduling is among the growing areas in Artificial Intelligence (AI) where mention of LLMs has gained popularity. Based on a comprehensive review of 126 papers, this paper investigates eight categories based on the unique applications of LLMs in addressing various aspects of planning problems: language translation, plan generation, model construction, multi-agent planning, interactive planning, heuristics optimization, tool integration, and brain-inspired planning. For each category, we articulate the issues considered and existing gaps. A critical insight resulting from our review is that the true potential of LLMs unfolds when they are integrated with traditional symbolic planners, pointing towards a promising neuro-symbolic approach. This approach effectively combines the generative aspects of LLMs with the precision of classical planning methods. By synthesizing insights from existing literature, we underline the potential of this integration to address complex planning challenges. Our goal is to encourage the ICAPS community to recognize the complementary strengths of LLMs and symbolic planners, advocating for a direction in automated planning that leverages these synergistic capabilities to develop more advanced and intelligent planning systems. We aim to keep the categorization of papers updated on https://ai4society.github.io/LLM-Planning-Viz/, a collaborative resource that allows researchers to contribute and add new literature to the categorization.

show abstract

Section: Language Translationmentioning

confidence: 99%

Section: Plan Generationmentioning

confidence: 99%

On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)

Pallagani,

Muppasani,

Roy

et al. 2024

ICAPS

View full text Add to dashboard Cite

show abstract

“…Related to this idea, we will also explore re-planning approaches if we encounter action failure or if action contexts have not been experienced before in the same vein of previous work [25]. Finally, we will also review methods to generalize knowledge and action contexts using semantic similarity [6], [28] to creatively extend concepts at the symbolic level or trajectories at the execution level to new object instances in the physical world.…”

Section: A Future Workmentioning

confidence: 99%

Task Planning with a Weighted Functional Object-Oriented Network

Paulius

Dong

Sun

2021

2021 IEEE International Conference on Robotics and Automation (ICRA)

Self Cite

View full text Add to dashboard Cite

Following work on joint object-action representations, functional object-oriented networks (FOON) were introduced as a knowledge graph representation for robots. Taking the form of a bipartite graph, a FOON contains symbolic (high-level) concepts useful to a robot's understanding of tasks and its environment for object-level planning. Prior to this paper, little has been done to demonstrate how task plans acquired from FOON via task tree retrieval can be executed by a robot, as the concepts in a FOON are too abstract for immediate execution. We propose a hierarchical task planning approach that translates a FOON graph into a PDDL-based representation of domain knowledge for manipulation planning. As a result of this process, a task plan can be acquired that a robot can execute from start to end, leveraging the use of action contexts and skills in the form of dynamic movement primitives (DMP). We demonstrate the entire pipeline from planning to execution using CoppeliaSim and show how learned action contexts can be extended to never-before-seen scenarios.

show abstract

“…Some experiments showed robotic chefs learning from human feedback on the taste of cooked dishes [24]. Some experiments with transcribing recipes into a set of actions [25] and robots cooking from recipes were done [26].…”

Section: Introductionmentioning

confidence: 99%

Recognition of Human Chef’s Intentions for Incremental Learning of Cookbook by Robotic Salad Chef

et al. 2023

View full text Add to dashboard Cite

Robotic chefs are a promising technology that can bring sizeable health and economic benefits when deployed ubiquitously. This deployment is hindered by the costly process of programming the robots to cook specific dishes while humans learn from observation or freely available videos. In this paper, we propose an algorithm that incrementally adds recipes to the robot's cookbook based on the visual observation of a human chef, enabling the easier and cheaper deployment of robotic chefs. A new recipe is added only if the current observation is substantially different than all recipes in the cookbook, which is decided by computing the similarity between the vectorizations of these two. The algorithm correctly recognizes known recipes in 93% of the demonstrations and successfully learned new recipes when shown, using off-the-shelf neural networks for computer vision. We show that videos and demonstrations are viable sources of data for robotic chef programming when extended to massive publicly available data sources like YouTube. INDEX TERMSComputer vision, hidden Markov model, learning by demonstration, robotic chef, salad chef.

show abstract

Approximate Task Tree Retrieval in a Knowledge Network for Robotic Cooking

Cited by 11 publications

References 33 publications

On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)

On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)

Task Planning with a Weighted Functional Object-Oriented Network

Recognition of Human Chef’s Intentions for Incremental Learning of Cookbook by Robotic Salad Chef

Contact Info

Product

Resources

About