A4T: Hierarchical Affordance Detection for Transparent Objects Depth Reconstruction and Manipulation

Jiang, Jianrong; Cao, Guanqun; Luo, Shan

doi:10.1109/lra.2022.3191231

Cited by 22 publications

(15 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As pointed out before, transparent objects face some further difficulties. There are some approaches using depth data, with some work on depth completion, as in [ 10 ], which uses affordance maps to reconstruct the depth data. Another approach is made by Xu et al [ 11 ].…”

Section: Related Workmentioning

confidence: 99%

“…This can distort the depth data quite a lot. There are different common approaches to tackle this problem: trying to reconstruct the depth data, as performed for example in [ 10 , 11 , 12 ], simply trying to solve the 6D pose estimation with stereo RGB images [ 13 ] or by just using single RGB images [ 14 , 15 ]. The latter is what our work focuses on.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Transparency-Aware Segmentation of Glass Objects to Train RGB-Based Pose Estimators

Weidenbach,

Laue,

Frese

2024

Sensors

View full text Add to dashboard Cite

Robotic manipulation requires object pose knowledge for the objects of interest. In order to perform typical household chores, a robot needs to be able to estimate 6D poses for objects such as water glasses or salad bowls. This is especially difficult for glass objects, as for these, depth data are mostly disturbed, and in RGB images, occluded objects are still visible. Thus, in this paper, we propose to redefine the ground-truth for training RGB-based pose estimators in two ways: (a) we apply a transparency-aware multisegmentation, in which an image pixel can belong to more than one object, and (b) we use transparency-aware bounding boxes, which always enclose whole objects, even if parts of an object are formally occluded by another object. The latter approach ensures that the size and scale of an object remain more consistent across different images. We train our pose estimator, which was originally designed for opaque objects, with three different ground-truth types on the ClearPose dataset. Just by changing the training data to our transparency-aware segmentation, with no additional glass-specific feature changes in the estimator, the ADD-S AUC value increases by 4.3%. Such a multisegmentation can be created for every dataset that provides a 3D model of the object and its ground-truth pose.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Transparency-Aware Segmentation of Glass Objects to Train RGB-Based Pose Estimators

Weidenbach,

Laue,

Frese

2024

Sensors

View full text Add to dashboard Cite

show abstract

“…However, those methods require multiple views of one object, which is not suitable for the case when the camera is fixed. To address this challenge, some other studies [3], [13], [14] focus on reconstructing the missing or noisy depth regions of transparent objects using a single RGB-D image. In [3], a global optimisation algorithm was adopted to reconstruct the depth values that are removed based on predicted object masks.…”

Section: A Transparent Object Graspingmentioning

confidence: 99%

“…In [13], a local implicit neural representation built on ray-voxel pairs was proposed to reconstruct depth information incorporated with an iterative self-correcting refinement model. In [14], an affordance-based depth reconstruction framework was proposed to facilitate the robotic manipulation of transparent objects.…”

Section: A Transparent Object Graspingmentioning

confidence: 99%

Where Shall I Touch? Vision-Guided Tactile Poking for Transparent Object Grasping

Jiang

Cao

Butterworth

et al. 2023

IEEE/ASME Trans. Mechatron.

Self Cite

View full text Add to dashboard Cite

Picking up transparent objects is still a challenging task for robots. The visual properties of transparent objects such as reflection and refraction make the current grasping methods that rely on camera sensing fail to detect and localise them. However, humans can handle the transparent object well by first observing its coarse profile and then poking an area of interest to get a fine profile for grasping. Inspired by this, we propose a novel framework of vision-guided tactile poking for transparent objects grasping. In the proposed framework, a segmentation network is first used to predict the horizontal upper regions named as poking regions, where the robot can poke the object to obtain a good tactile reading while leading to minimal disturbance to the object's state. A poke is then performed with a high-resolution GelSight tactile sensor. Given the local profiles improved with the tactile reading, a heuristic grasp is planned for grasping the transparent object. To mitigate the limitations of real-world data collection and labelling for transparent objects, a large-scale realistic synthetic dataset was constructed. Extensive experiments demonstrate that our proposed segmentation network can predict the potential poking region with a high mean Average Precision (mAP) of 0.360, and the vision-guided tactile poking can enhance the grasping success rate significantly from 38.9% to 85.2%. Thanks to its simplicity, our proposed approach could also be adopted by other force or tactile sensors and could be used for grasping of other challenging objects. All the materials used in this paper are available at https://sites.google.com/view/tactilepoking.

show abstract

“…Table 1 summarizes recent datasets for transparent object recognition. As mainstream, most researchers were focused on the RGB-based datasets (Bashkirova et al, 2022; Chen et al, 2018, 2022; Dai et al, 2022; Fang et al, 2022; Jiang et al, 2022; Jiang and Shan Ll, 2022b; Lin et al, 2021; Liu et al, 2020, 2021; Mei et al, 2020; Proença and Simoes, 2020; Sajjan et al, 2020; Xie et al, 2020; Xu et al, 2022). These datasets have necessitated background clutters using additional artifacts to overcome the attribute of the transparent objects heavily dependent on the background.…”

Section: Related Workmentioning

confidence: 99%

TRansPose: Large-scale multispectral dataset for transparent object

Kim,

Jeon,

Jung

et al. 2023

The International Journal of Robotics Research

View full text Add to dashboard Cite

Transparent objects are encountered frequently in our daily lives, yet recognizing them poses challenges for conventional vision sensors due to their unique material properties, not being well perceived from RGB or depth cameras. Overcoming this limitation, thermal infrared cameras have emerged as a solution, offering improved visibility and shape information for transparent objects. In this paper, we present TRansPose, the first large-scale multispectral dataset that combines stereo RGB-D, thermal infrared (TIR) images, and object poses to promote transparent object research. The dataset includes 99 transparent objects, encompassing 43 household items, 27 recyclable trashes, 29 chemical laboratory equivalents, and 12 non-transparent objects. It comprises a vast collection of 333,819 images and 4,000,056 annotations, providing instance-level segmentation masks, ground-truth poses, and completed depth information. The data was acquired using an FLIR A65 thermal infrared camera, two Intel RealSense L515 RGB-D cameras, and a Franka Emika Panda robot manipulator. Spanning 87 sequences, TRansPose covers various challenging real-life scenarios, including objects filled with water, diverse lighting conditions, heavy clutter, non-transparent or translucent containers, objects in plastic bags, and multi-stacked objects. Supplementary material can be accessed from the following link: https://sites.google.com/view/transpose-dataset .

show abstract

A4T: Hierarchical Affordance Detection for Transparent Objects Depth Reconstruction and Manipulation

Cited by 22 publications

References 26 publications

Transparency-Aware Segmentation of Glass Objects to Train RGB-Based Pose Estimators

Transparency-Aware Segmentation of Glass Objects to Train RGB-Based Pose Estimators

Where Shall I Touch? Vision-Guided Tactile Poking for Transparent Object Grasping

TRansPose: Large-scale multispectral dataset for transparent object

Contact Info

Product

Resources

About