Virtual Visual Servoing for Real-Time Robot Pose Estimation

Gratal, Xavi; Romero, Javier; Kragić, Danica

doi:10.3182/20110828-6-it-1002.02970

Cited by 19 publications

(14 citation statements)

References 17 publications

(18 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also, false information or distracting information from one module is expected to impair action recognition and simulation capabilities in the connected modules. On the robotics side, related techniques were applied using virtual visual servoing for object tracking (Comport et al, 2006) and for improving the pose estimates of a robot (Gratal et al, 2011). Our model offers both generative, visual servoing options and temporal motion predictions and inference-based, action recognition capabilities.…”

Section: Discussionmentioning

confidence: 99%

Just Imagine! Learning to Emulate and Infer Actions with a Stochastic Generative Architecture

Schrodt

Butz

2016

Front. Robot. AI

View full text Add to dashboard Cite

Theories on embodied cognition emphasize that our mind develops by processing and inferring structures given the encountered bodily experiences. Here, we propose a distributed neural network architecture that learns a stochastic generative model from experiencing bodily actions. Our modular system learns from various manifolds of action perceptions in the form of (i) relative positional motion of the individual body parts, (ii) angular motion of joints, and (iii) relatively stable top-down action identities. By Hebbian learning, this information is spatially segmented in separate neural modules that provide embodied state codes and temporal predictions of the state progression inside and across the modules. The network is generative in space and time, thus being able to predict both, missing sensory information and next sensory information. We link the developing encodings to visuomotor and multimodal representations that appear to be involved in action observation. Our results show that the system learns to infer action types and motor codes from partial sensory information by emulating observed actions with the own developing body model. We further evaluate the generative capabilities by showing that the system is able to generate internal imaginations of the learned types of actions without sensory stimulation, including visual images of the actions. The model highlights the important roles of motor cognition and embodied simulation for bootstrapping action understanding capabilities. We conclude that stochastic generative models appear very suitable for both, generating goal-directed actions and predicting observed visuomotor trajectories and action goals.

show abstract

Section: Discussionmentioning

confidence: 99%

Just Imagine! Learning to Emulate and Infer Actions with a Stochastic Generative Architecture

Schrodt

Butz

2016

Front. Robot. AI

View full text Add to dashboard Cite

show abstract

“…This is fundamental for grasping objects. The first breakthrough dealing with humanoid robots and specifically tracking of an anthropomorphic hand are relatively recent, dating back to 2006 [29]- [32].…”

Section: Related Workmentioning

confidence: 99%

“…In [29] the authors proposed a novel framework based on the Virtual Visual Servoing (VVS) paradigm, first introduced in [33], [34]. 3D CAD models are used within a rendering engine to virtually create the hand-effector as if it had been seen by a camera using information provided by the direct kinematics.…”

Section: Related Workmentioning

confidence: 99%

Visual end-effector tracking using a 3D model-aided particle filter for humanoid robot platforms

Fantacci

Pattacini

Tikhanoff

et al. 2017

2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Abstract-This paper addresses recursive markerless estimation of a robot's end-effector using visual observations from its cameras. The problem is formulated into the Bayesian framework and addressed using Sequential Monte Carlo (SMC) filtering. We use a 3D rendering engine and Computer Aided Design (CAD) schematics of the robot to virtually create images from the robot's camera viewpoints. These images are then used to extract information and estimate the pose of the end-effector. To this aim, we developed a particle filter for estimating the position and orientation of the robot's end-effector using the Histogram of Oriented Gradient (HOG) descriptors to capture robust characteristic features of shapes in both cameras and rendered images. We implemented the algorithm on the iCub humanoid robot and employed it in a closed-loop reaching scenario. We demonstrate that the tracking is robust to clutter, allows compensating for errors in the robot kinematics and servoing the arm in closed loop using vision.

show abstract

“…However, in both works the hand orientation is neglected -only the position of the hand is learned. The work by Gratal et al (2011) proposes a 3D-model based approach and an edge based error function to estimate the pose of the Schunk Dexterous Hand. This method is similar to ours as they exploit also graphics acceleration techniques and an edge-based approach.…”

Section: Related Workmentioning

confidence: 99%

Online Body Schema Adaptation Based on Internal Mental Simulation and Multisensory Feedback

2016

View full text Add to dashboard Cite

In this paper, we describe a novel approach to obtain automatic adaptation of the robot body schema and to improve the robot perceptual and motor skills based on this body knowledge. Predictions obtained through a mental simulation of the body are combined with the real sensory feedback to achieve two objectives simultaneously: body schema adaptation and markerless 6D hand pose estimation. The body schema consists of a computer graphics simulation of the robot, which includes the arm and head kinematics (adapted online during the movements) and an appearance model of the hand shape and texture. The mental simulation process generates predictions on how the hand will appear in the robot camera images, based on the body schema and the proprioceptive information (i.e., motor encoders). These predictions are compared to the actual images using sequential Monte Carlo techniques to feed a particle-based Bayesian estimation method to estimate the parameters of the body schema. The updated body schema will improve the estimates of the 6D hand pose, which is then used in a closed-loop control scheme (i.e., visual servoing), enabling precise reaching. We report experiments with the iCub humanoid robot that support the validity of our approach. A number of simulations with precise ground-truth were performed to evaluate the estimation capabilities of the proposed framework. Then, we show how the use of high-performance GPU programing and an edge-based algorithm for visual perception allow for real-time implementation in real-world scenarios.

show abstract

Virtual Visual Servoing for Real-Time Robot Pose Estimation

Cited by 19 publications

References 17 publications

Just Imagine! Learning to Emulate and Infer Actions with a Stochastic Generative Architecture

Just Imagine! Learning to Emulate and Infer Actions with a Stochastic Generative Architecture

Visual end-effector tracking using a 3D model-aided particle filter for humanoid robot platforms

Online Body Schema Adaptation Based on Internal Mental Simulation and Multisensory Feedback

Contact Info

Product

Resources

About