GraspNeRF: Multiview-based 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRF

Qiyu, Dai,; Zhu, Yan; Geng, Yiran; Ciyu, Ruan,; Zhang, Jiazhao; Wang, He

doi:10.48550/arxiv.2210.06575

Cited by 3 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…NeRF predicts the RGB color and density of a point in a scene so that an image from an arbitrary viewpoint can be rendered. This property enables pose estimation [1,30,31,44] based on the photometric loss between the observed image and the rendered image, or manipulation of tricky objects [5,12,14,25,29]. A pretrained NeRF can also work as a virtual simulator, in which a robot can plan its trajectory [1] or can be used to train an action policy for the real-world [6].…”

Section: Related Workmentioning

confidence: 99%

Renderable Neural Radiance Map for Visual Navigation

Kwon¹,

Park²,

Oh³

2023

Preprint

View full text Add to dashboard Cite

We propose a novel type of map for visual navigation, a renderable neural radiance map (RNR-Map), which is designed to contain the overall visual information of a 3D environment. The RNR-Map has a grid form and consists of latent codes at each pixel. These latent codes are embedded from image observations, and can be converted to the neural radiance field which enables image rendering given a camera pose. The recorded latent codes implicitly contain visual information about the environment, which makes the RNR-Map visually descriptive. This visual information in RNR-Map can be a useful guideline for visual localization and navigation. We develop localization and navigation frameworks that can effectively utilize the RNR-Map. We evaluate the proposed frameworks on camera tracking, visual localization, and image-goal navigation. Experimental results show that the RNR-Map-based localization framework can find the target location based on a single query image with fast speed and competitive accuracy compared to other baselines. Also, this localization framework is robust to environmental changes, and even finds the most visually similar places when a query image from a different environment is given. The proposed navigation framework outperforms the existing image-goal navigation methods in difficult scenarios, under odometry and actuation noises. The navigation framework shows 65.7% success rate in curved scenarios of the NRNS [23] dataset, which is an improvement of 18.6% over the current state-of-the-art. Project page: https: //rllab-snu.github.io/projects/RNR-Map/

show abstract

Section: Related Workmentioning

confidence: 99%

Renderable Neural Radiance Map for Visual Navigation

Kwon¹,

Park²,

Oh³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…However, these approaches often require retraining the NeRF model before each grasp to update the environment states. GraspN-eRF [20] addresses this constraint by proposing a generalisable NeRF that is free from per-scene optimisation. Nevertheless, GraspNeRF is not object-centric and thus cannot interpret the scene at the object level.…”

Section: Related Workmentioning

confidence: 99%

APEX: Unsupervised, Object-Centric Scene Segmentation and Tracking for Robot Manipulation

Jones

Engelcke

et al. 2021

2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Recent advances in unsupervised learning for object detection, segmentation, and tracking hold significant promise for applications in robotics. A common approach is to frame these tasks as inference in probabilistic latent-variable models. In this paper, however, we show that the current state-of-the-art struggles with visually complex scenes such as typically encountered in robot manipulation tasks. We propose APEX, a new latent-variable model which is able to segment and track objects in more realistic scenes featuring objects that vary widely in size and texture, including the robot arm itself. This is achieved by a principled mask normalisation algorithm and a high-resolution scene encoder. To evaluate our approach, we present results on the real-world Sketchy dataset. This dataset, however, does not contain ground truth masks and object IDs for a quantitative evaluation. We thus introduce the Panda Pushing Dataset (P2D) which shows a Panda arm interacting with objects on a table in simulation and which includes groundtruth segmentation masks and object IDs for tracking. In both cases, APEX comprehensively outperforms the current state-ofthe-art in unsupervised object segmentation and tracking. We demonstrate the efficacy of our segmentations for robot skill execution on an object arrangement task, where we also achieve the best or comparable performance among all the baselines.

show abstract

“…Representatives are various glass objects, e.g., glass walls or glass flasks, and transparent or translucent plastic parts, e.g., clear orthodontic aligners or car headlights. Typical areas of application are as follows: (a) human-robot interactions, e.g., for confidential detection of visually uncooperative objects [1]; (b) autonomous robot navigation, e.g., collision prevention of glass walls; (c) laboratory automation, e.g., for grasping visually uncooperative objects [2][3][4][5]; (d) medical section, e.g., 3D reconstruction of clear orthodontic aligners; (e) autonomous waste sorting and recycling, and (f) augmented reality [6]. In these use cases, there are two main tasks:…”

Section: Introductionmentioning

confidence: 99%

“…• Accurately estimating the depth of optically uncooperative objects. This includes accurate and reliable depth estimates, also known as deep depth completion [2,[12][13][14], 3D reconstruction methods [3,15,16], and stereo vision [17][18][19].…”

mentioning

confidence: 99%

TranSpec3D: A Novel Measurement Principle to Generate A Non-Synthetic Data Set of Transparent and Specular Surfaces without Object Preparation

Junger,

Speck,

Landmann

et al. 2023

Sensors

View full text Add to dashboard Cite

Estimating depth from images is a common technique in 3D perception. However, dealing with non-Lambertian materials, e.g., transparent or specular, is still nowadays an open challenge. However, to overcome this challenge with deep stereo matching networks or monocular depth estimation, data sets with non-Lambertian objects are mandatory. Currently, only few real-world data sets are available. This is due to the high effort and time-consuming process of generating these data sets with ground truth. Currently, transparent objects must be prepared, e.g., painted or powdered, or an opaque twin of the non-Lambertian object is needed. This makes data acquisition very time consuming and elaborate. We present a new measurement principle for how to generate a real data set of transparent and specular surfaces without object preparation techniques, which greatly reduces the effort and time required for data collection. For this purpose, we use a thermal 3D sensor as a reference system, which allows the 3D detection of transparent and reflective surfaces without object preparation. In addition, we publish the first-ever real stereo data set, called TranSpec3D, where ground truth disparities without object preparation were generated using this measurement principle. The data set contains 110 objects and consists of 148 scenes, each taken in different lighting environments, which increases the size of the data set and creates different reflections on the surface. We also show the advantages and disadvantages of our measurement principle and data set compared to the Booster data set (generated with object preparation), as well as the current limitations of our novel method.

show abstract

GraspNeRF: Multiview-based 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRF

Cited by 3 publications

References 0 publications

Renderable Neural Radiance Map for Visual Navigation

Renderable Neural Radiance Map for Visual Navigation

APEX: Unsupervised, Object-Centric Scene Segmentation and Tracking for Robot Manipulation

TranSpec3D: A Novel Measurement Principle to Generate A Non-Synthetic Data Set of Transparent and Specular Surfaces without Object Preparation

Contact Info

Product

Resources

About