Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds

Thach, Bao; Cho, Brian Y.; Kuntz, Alan; Hermans, Tucker

doi:10.1109/icra46639.2022.9812215

Cited by 25 publications

(14 citation statements)

References 63 publications

(98 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Policy Learning from Point Clouds or Flow for Robotic Manipulation Learning from point clouds has been explored in grasping [19,27], in-hand manipulation (by voxelizing) [18], visual navigation [28], and shaping 3D deformables [29]. Our work differs in that we study tasks that involve manipulating a tool in 3D space from point cloud observations, and where we use tool flow as the action representation to improve learning.…”

Section: Related Workmentioning

confidence: 99%

ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds

Seita¹,

Wang²,

Shetty³

et al. 2022

Preprint

View full text Add to dashboard Cite

Point clouds are a widely available and canonical data modality which convey the 3D geometry of a scene. Despite significant progress in classification and segmentation from point clouds, policy learning from such a modality remains challenging, and most prior works in imitation learning focus on learning policies from images or state information. In this paper, we propose a novel framework for learning policies from point clouds for robotic manipulation with tools. We use a novel neural network, ToolFlowNet, which predicts dense perpoint flow on the tool that the robot controls, and then uses the flow to derive the transformation that the robot should execute. We apply this framework to imitation learning of challenging deformable object manipulation tasks with continuous movement of tools, including scooping and pouring, and demonstrate significantly improved performance over baselines which do not use flow. We perform 50 physical scooping experiments with ToolFlowNet and attain 82% scooping success. See https://tinyurl.com/toolflownet for supplementary material.

show abstract

Section: Related Workmentioning

confidence: 99%

ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds

Seita¹,

Wang²,

Shetty³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Other works focused on environments with better dynamics interactions between the da Vinci tools and soft tissues [12,13]. These works have been followed by more recent works presented in [5,14] where there has been improvement in terms of environment dynamics and task characterisation.…”

Section: Rl In Surgical Roboticsmentioning

confidence: 99%

Learning intraoperative organ manipulation with context-based reinforcement learning

et al. 2022

View full text Add to dashboard Cite

Purpose Automation of sub-tasks during robotic surgery is challenging due to the high variability of the surgical scenes intra- and inter-patients. For example, the pick and place task can be executed different times during the same operation and for distinct purposes. Hence, designing automation solutions that can generalise a skill over different contexts becomes hard. All the experiments are conducted using the Pneumatic Attachable Flexible (PAF) rail, a novel surgical tool designed for robotic-assisted intraoperative organ manipulation. Methods We build upon previous open-source surgical Reinforcement Learning (RL) training environment to develop a new RL framework for manipulation skills, rlman. In rlman, contextual RL agents are trained to solve different aspects of the pick and place task using the PAF rail system. rlman is implemented to support both low- and high-dimensional state information to solve surgical sub-tasks in a simulation environment. Results We use rlman to train state of the art RL agents to solve four different surgical sub-tasks involving manipulation skills using the PAF rail. We compare the results with state-of-the-art benchmarks found in the literature. We evaluate the ability of the agent to be able to generalise over different aspects of the targeted surgical environment. Conclusion We have shown that the rlman framework can support the training of different RL algorithms for solving surgical sub-task, analysing the importance of context information for generalisation capabilities. We are aiming to deploy the trained policy on the real da Vinci using the dVRK and show that the generalisation of the trained policy can be transferred to the real world.

show abstract

“…Recently, visionbased deformable object manipulation has been widely investigated. The robot is supposed to infer a sequence of manipulation actions from solely visual observations to manipulate a deformable object into a prescribed goal configuration [3], [4]. Early works focus on learning task-specific manipulation skills and have limited application.…”

Section: Introductionmentioning

confidence: 99%

Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention

Xia

Wang

et al. 2023

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Vision-based deformable object manipulation is a challenging problem in robotic manipulation, requiring a robot to infer a sequence of manipulation actions leading to the desired state from solely visual observations. Most previous works address this problem in a goal-conditioned way and adapt the goal image to specify a task, which is not practical or efficient. Thus, we adapted natural language specification and proposed a language-conditioned deformable object manipulation policy learning framework. We first design a unified Transformer-based architecture to understand multimodal data and output picking and placing action. Besides, we have introduced the visible connectivity graph to tackle nonlinear dynamics and complex configuration of the deformable object in the manipulation process. Both simulated and real experiments have demonstrated that the proposed method is general and effective in language-conditioned deformable object manipulation policy learning. Our method achieves much higher success rates on various language-conditioned deformable object manipulation tasks (87.3% on average) than the state-of-the-art method in simulation experiments. Besides, our method is much lighter and has a 75.6% shorter inference time than state-of-the-art methods. We also demonstrate that our method performs well in real-world applications. Supplementary videos can be found at https://sites.google. com/view/language-deformable.

show abstract

Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds

Cited by 25 publications

References 63 publications

ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds

ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds

Learning intraoperative organ manipulation with context-based reinforcement learning

Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention

Contact Info

Product

Resources

About