Providing synthetic views for teleoperation using visual pose tracking in multiple cameras

Abstract

This paper describes a visual tool for teleoperative experimentation involving remote manipulation and contact tasks. Using modest hardware, it recovers in real-time the pose of moving polyhedral objects, and presents a synthetic view of the scene to the teleoperator using any chosen viewpoint and viewing direction. The method of line tracking introduced by Harris is extended to multiple calibrated cameras, and afforced by robust methods and iterative filtering. Experiments are reported which determine the static and dynamic performance of the vision system, and its use in teleoperation is illustrated in two experiments, a peg in hole manipulation task and an impact control task.

Modelling an arm's spatially correlated biases: an application to camera calibration for teleoperation

“…The camera calibration method has been implemented using the slave Stäubli Puma arm of the Oxford teleoperation system [21,22]. Two practical issues of particular importance are, first, the accuracy of location of the end-effector in the images where the cameras have quite different viewpoints and, second, the robustness of location.…”

Operator matching during visually aided teleoperation

“…In more recent work, rather than using the video feed directly, a model-based machine vision system has been interposed between camera and display to recover the pose of known objects and to track them over time [4]. An artificial view is generated from the pose and presented to the operator, who is able to choose a convenient viewpoint for various phases of the task, be it alignment, insertion, or contact.…”

“…[4][5][6][7][8]). In [4], for example, the model is used to predict pose in the newly captured image as a starting point for an iterative refinement which at convergence gives the current pose raw measurement. The pose measurement is then used to update the modelled pose state in a Kalman filter, as illustrated in Fig.…”

“…This is of concern because methods of model-based trackers typically use a dynamical model, usually a constant velocity model, to reduce the need for expensive image search and to increase robustness (e.g. [4][5][6][7][8]). In [4], for example, the model is used to predict pose in the newly captured image as a starting point for an iterative refinement which at convergence gives the current pose raw measurement.…”

A system to navigate a robot into a ship structure

“…The control points are searched normal to the edge direction and then used for pose estimation. This work is improved [25] by adding a median filter to detect outliers for fitting the line and the pose. They report the improvements using black-and-white objects with little clutter in the background.…”