Video Object Segmentation-based Visual Servo Control and Object Depth Estimation on a Mobile Robot

Griffin, Brent; Florence, Victoria; Corso, Jason J.

doi:10.1109/wacv45572.2020.9093335

Cited by 22 publications

(25 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Results demonstrate the efficiency of our approach, and the ablation study shows how our approach outperforms selected baselines, achieving a grasping accuracy of 83 % on previously unseen semi-compliant objects. This is on par with or better than recent state-of-the-art methods [14], [20], [22], [31]. To the best of our knowledge, this work is the first to use a GAN in combination with VS for the transfer of a DRL grasping policy to a real robot.…”

Section: Discussionmentioning

confidence: 75%

See 1 more Smart Citation

Grasping Unknown Objects by Coupling Deep Reinforcement Learning, Generative Adversarial Networks, and Visual Servoing

Pedersen

Misimi

Chaumette

2020

2020 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

In this paper, we propose a novel approach for transferring a deep reinforcement learning (DRL) grasping agent from simulation to a real robot, without fine tuning in the real world. The approach utilises a CycleGAN to close the reality gap between the simulated and real environments, in a reverse real-to-sim manner, effectively "tricking" the agent into believing it is still in the simulator. Furthermore, a visual servoing (VS) grasping task is added to correct for inaccurate agent gripper pose estimations derived from deep learning. The proposed approach is evaluated by means of real grasping experiments, achieving a success rate of 83 % on previously seen objects, and the same success rate for previously unseen, semi-compliant objects. The robustness of the approach is demonstrated by comparing it with two baselines, DRL plus CycleGAN, and VS only. The results clearly show that our approach outperforms both baselines.

show abstract

Section: Discussionmentioning

confidence: 75%

“…VS is a collection of closed-loop robotic control techniques based on visual feedback [30]. This paper uses a geometric VS task dependent on a segmentation of the scene, similar to [31], however in this work the segmentation network is trained with the domain adaption network, instead of using a separate video object segmentation network.…”

Section: Introductionmentioning

confidence: 99%

Grasping Unknown Objects by Coupling Deep Reinforcement Learning, Generative Adversarial Networks, and Visual Servoing

Pedersen

Misimi

Chaumette

2020

2020 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

show abstract

“…To bridge the gap between 3D applications and progress in video object segmentation, in recent work [14], we developed a method of video object segmentation-based visual servo control, object depth estimation, and mobile robot Object detectors can reliably place bounding boxes on target objects in a variety of settings. Given a sequence of bounding boxes and camera movement distances between observations (e.g., from robot kinematics, top), our network (DBox) estimates each object's depth (bottom).…”

Section: Introductionmentioning

confidence: 99%

“…grasping using a single RGB camera. For object depth estimation, specifically, we used the optical expansion [22,46] of segmentation masks with z-axis camera motion to analytically solve for depth [14,]. In subsequent work [15], we introduced the first learning-based method (ODN) and benchmark dataset for estimating Object Depth via Motion and Segmentation (ODMS), which includes test sets in robotics and driving.…”

Section: Introductionmentioning

confidence: 99%

“…Motivated by these developments [14,15], this paper ad-dresses the problem of estimating the depth of objects using uncalibrated camera motion and bounding boxes from object detection (see Figure 1), which has many advantages. First, a bounding box has only four parameters, can be processed quickly with few resources, and has less domain-specific features than an RGB image or segmentation mask.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning Object Depth from Camera Motion and Video Object Segmentation

Griffin

Corso

2020

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

This paper addresses the problem of learning to estimate the depth of detected objects given some measurement of camera motion (e.g., from robot kinematics or vehicle odometry). We achieve this by 1) designing a recurrent neural network (DBox) that estimates the depth of objects using a generalized representation of bounding boxes and uncalibrated camera movement and 2) introducing the Object Depth via Motion and Detection Dataset (ODMD). ODMD training data are extensible and configurable, and the ODMD benchmark includes 21,600 examples across four validation and test sets. These sets include mobile robot experiments using an end-effector camera to locate objects from the YCB dataset and examples with perturbations added to camera motion or bounding box data. In addition to the ODMD benchmark, we evaluate DBox in other monocular application domains, achieving state-of-the-art results on existing driving and robotics benchmarks and estimating the depth of objects using a camera phone.

show abstract

Multi-modal scene graph inspired policy for visual navigation

He,

Zhou,

Tian

2024

J Supercomput

View full text Add to dashboard Cite

Visual navigation needs the agent locate the given target with visual perception. To enable robots to effectively execute tasks, combining large language models (LLMs) with multi-modal inputs in navigation is necessary. While LLMs offer rich semantic knowledge, they lack specific real-world information and real-time interaction capabilities. This paper introduces a Multi-modal Scene Graph (MMSG) navigation framework that aligns LLMs with visual perception models to predict next steps. Firstly, a multi-modal scene dataset is constructed, containing triplets of object-relations-target words. We provide target words and lists of existing objects in the scene to generate a large number of instructions and corresponding action plans for GPT$$-$$ - 3.5. The generated data is then utilized for pre-train LLM for path planning. During inference, we discover objects in the scene by extending the DETR visual object detector to multi-view RGB image collected from different reachable positions. Experimental results show that path planning generated from MMSG outperforms state-of-the-art methods, indicating its feasibility in complex environments. We evaluate our methods on the ProTHOR dataset and show superior navigation performance.

show abstract

Video Object Segmentation-based Visual Servo Control and Object Depth Estimation on a Mobile Robot

Cited by 22 publications

References 51 publications

Grasping Unknown Objects by Coupling Deep Reinforcement Learning, Generative Adversarial Networks, and Visual Servoing

Grasping Unknown Objects by Coupling Deep Reinforcement Learning, Generative Adversarial Networks, and Visual Servoing

Learning Object Depth from Camera Motion and Video Object Segmentation

Multi-modal scene graph inspired policy for visual navigation

Contact Info

Product

Resources

About