A framework for tool cognition in robots without prior tool learning or observation

Tee, Keng Peng; Cheong, Samuel; Li, Jun; Ganesh, Gowrishankar

doi:10.1038/s42256-022-00500-9

Cited by 5 publications

(5 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While the above taxonomies are based on animal studies, Tee et al. (2018 , 2022) proposed a categorization based on default usages of tools in robotics, and identified three types of tools: category-I tools that “help to amplify/augment certain kinematic or dynamic aspects of functions that are already in an agents repertoire.” (p. 6439), category-II tools that are similar to category-I tools but “require actions different from what the agent would have performed, without the tool, to achieve these functions.” (p. 6439), and category-III “provide new functions that a human cannot perform without a tool.” (p. 6440) As an example, they categorized a vacuum cleaner as a Category-III tool because a robot cannot perform a cleaning task without this tool. However, a vacuum cleaner can be used as a rake to reach objects or as a hammer to hit objects in other contexts.…”

Section: A Taxonomy Of Robot Tool Usementioning

confidence: 99%

“…While the above taxonomies are based on animal studies, Tee et al (2018Tee et al ( , 2022 proposed a categorization based on default usages of tools in robotics, and identified three types of tools: category-I tools that "help to amplify/augment certain kinematic or dynamic aspects of functions that are already in an agents repertoire. " (p. 6439), category-II tools that are similar to category-I tools but "require actions different from what the agent would have performed, without the tool, to achieve these functions. "…”

Section: Taxonomy Of Robot Tool Usementioning

confidence: 99%

“…Therefore, other researchers investigated algorithms that transfer learned skills more broadly and demonstrated with multiple tool use tasks. Tee et al (2018Tee et al ( , 2022 matched the point cloud of unseen tools to the point cloud of the endeffector and arms of the robot to obtain the usage of the tools. The kPAM/kPAM 2.0 (Manuelli et al, 2019;Gao and Tedrake, 2021) used keypoints on the tools to represent shared global shape of the category of tools, and tool use skills were inferred from these keypoints.…”

Section: Transferable Tool Usementioning

confidence: 99%

See 2 more Smart Citations

Robot tool use: A survey

Qin

Brawer²,

Scassellati³

2023

Front. Robot. AI

View full text Add to dashboard Cite

Using human tools can significantly benefit robots in many application domains. Such ability would allow robots to solve problems that they were unable to without tools. However, robot tool use is a challenging task. Tool use was initially considered to be the ability that distinguishes human beings from other animals. We identify three skills required for robot tool use: perception, manipulation, and high-level cognition skills. While both general manipulation tasks and tool use tasks require the same level of perception accuracy, there are unique manipulation and cognition challenges in robot tool use. In this survey, we first define robot tool use. The definition highlighted the skills required for robot tool use. The skills coincide with an affordance model which defined a three-way relation between actions, objects, and effects. We also compile a taxonomy of robot tool use with insights from animal tool use literature. Our definition and taxonomy lay a theoretical foundation for future robot tool use studies and also serve as practical guidelines for robot tool use applications. We first categorize tool use based on the context of the task. The contexts are highly similar for the same task (e.g., cutting) in non-causal tool use, while the contexts for causal tool use are diverse. We further categorize causal tool use based on the task complexity suggested in animal tool use studies into single-manipulation tool use and multiple-manipulation tool use. Single-manipulation tool use are sub-categorized based on tool features and prior experiences of tool use. This type of tool may be considered as building blocks of causal tool use. Multiple-manipulation tool use combines these building blocks in different ways. The different combinations categorize multiple-manipulation tool use. Moreover, we identify different skills required in each sub-type in the taxonomy. We then review previous studies on robot tool use based on the taxonomy and describe how the relations are learned in these studies. We conclude with a discussion of the current applications of robot tool use and open questions to address future robot tool use.

show abstract

Section: A Taxonomy Of Robot Tool Usementioning

confidence: 99%

Section: Taxonomy Of Robot Tool Usementioning

confidence: 99%

Section: Transferable Tool Usementioning

confidence: 99%

See 1 more Smart Citation

Robot tool use: A survey

Qin

Brawer²,

Scassellati³

2023

Front. Robot. AI

View full text Add to dashboard Cite

show abstract

“…In contrast, observing the above from the perspective of robots as agents with embodied intelligence, achieving these physical interactions is currently far from trivial 5,6 and solving complex sequential tasks over a long horizon remains an ongoing challenge 7,8 . Notably, a task as simple as retrieving a glass from a shelf, pouring in water and placing it onto a table may seem trivial, but from an embodied intelligence perspective remains considerably challenging.…”

mentioning

confidence: 99%

“…We studied a set of seven specialized manipulation skills that are common in daily life and can be combined to create a higher level of manipulation skills. These specialized skills included (1) pushing a button, (2) pushing, (3) picking and inserting, (4) picking and placing, (5) rotating-opening, (6) picking and dropping and (7) pulling-opening. Unlike conventional planning methods or state machines, ROMAN exhibits adaptability in (1) randomized task sequences, (2) generalization outside demonstrated cases and (3) recovery and robustness against local minima.…”

mentioning

confidence: 99%

Hybrid hierarchical learning for solving complex sequential tasks using the robotic manipulation network ROMAN

Triantafyllidis,

Acero,

Liu

et al. 2023

Nat Mach Intell

View full text Add to dashboard Cite

Solving long sequential tasks remains a non-trivial challenge in the field of embodied artificial intelligence. Enabling a robotic system to perform diverse sequential tasks with a broad range of manipulation skills is a notable open problem and continues to be an active area of research. In this work, we present a hybrid hierarchical learning framework, the robotic manipulation network ROMAN, to address the challenge of solving multiple complex tasks over long time horizons in robotic manipulation. By integrating behavioural cloning, imitation learning and reinforcement learning, ROMAN achieves task versatility and robust failure recovery. It consists of a central manipulation network that coordinates an ensemble of various neural networks, each specializing in different recombinable subtasks to generate their correct in-sequence actions, to solve complex long-horizon manipulation tasks. Our experiments show that, by orchestrating and activating these specialized manipulation experts, ROMAN generates correct sequential activations accomplishing long sequences of sophisticated manipulation tasks and achieving adaptive behaviours beyond demonstrations, while exhibiting robustness to various sensory noises. These results highlight the significance and versatility of ROMAN’s dynamic adaptability featuring autonomous failure recovery capabilities, and underline its potential for various autonomous manipulation tasks that require adaptive motor skills.

show abstract

Tool use acquisition induces a multifunctional interference effect during object processing: evidence from the sensorimotor mu rhythm

Foerster

2023

Exp Brain Res

View full text Add to dashboard Cite

A framework for tool cognition in robots without prior tool learning or observation

Cited by 5 publications

References 62 publications

Robot tool use: A survey

Robot tool use: A survey

Hybrid hierarchical learning for solving complex sequential tasks using the robotic manipulation network ROMAN

Tool use acquisition induces a multifunctional interference effect during object processing: evidence from the sensorimotor mu rhythm

Contact Info

Product

Resources

About