Precise Multi-Modal In-Hand Pose Estimation using Low-Precision Sensors for Robotic Assembly

Drigalski, Felix von; Hayashi, Kennosuke; Huang, Yifei; Yonetani, Ryo; Hamaya, Masashi; Tanaka, Kazutoshi; Ijiri, Yoshihisa

doi:10.1109/icra48506.2021.9561222

Cited by 28 publications

(17 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The first two actions (Touch and Look) are described in [5], [6]. The Look action requires either a calibrated camera or a calibration geometry in the image, while the Touch action requires the position of at least one calibrated support surface (and optionally one edge) in the environment.…”

Section: Proposed Methodsmentioning

confidence: 99%

“…Each action reduces the pose uncertainty in different ways. The calculation of the Touch and Look action is described in our previous work [5], [6]. For the three extrinsic manipulations, we assume that the pose distribution after the action can be approximated by the representation in §III-A.…”

Section: B Calculating Pose Uncertaintymentioning

confidence: 99%

“…Previously, we developed an in-hand pose estimation method in which the robot touches the environment with the grasped part to reduce the uncertainty of the part's pose to sub-mm precision [5] (the Touch action). We extended this by also evaluating the effect of a camera view on the object pose uncertainty (the Look action) and allowing a combination of both actions [6]. However, these methods were limited: the user had to define the action sequence manually, and the actions required either slow motions and a force sensor (for the Touch action), or careful camera calibration (for the Look action).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Uncertainty-Aware Manipulation Planning Using Gravity and Environment Geometry

Drigalski

Kasaura

Beltran-Hernandez

et al. 2022

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

Factory automation robot systems often depend on specially-made jigs that precisely position each part, which increases the system's cost and limits flexibility. We propose a method to determine the 3D pose of an object with high precision and confidence, using only parallel robotic grippers and no parts-specific jigs. Our method automatically generates a sequence of actions that ensures that the real-world position of the physical object matches the system's assumed pose to sub-mm precision. Furthermore, we propose the use of "extrinsic" actions, which use gravity, the environment and the gripper geometry to significantly reduce or even eliminate the uncertainty about an object's pose. We show in simulated and real-robot experiments that our method outperforms our previous work, at success rates over 95%. The source code was made public at github.com/omron-sinicx.

show abstract

Section: Proposed Methodsmentioning

confidence: 99%

Section: B Calculating Pose Uncertaintymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Uncertainty-Aware Manipulation Planning Using Gravity and Environment Geometry

Drigalski

Kasaura

Beltran-Hernandez

et al. 2022

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Heterogeneous sensor modalities hold the promise of providing more informative feedback for solving manipulation tasks than uni-modal approaches. Specifically, visual and tactile data [13] have been utilized in object cluttering [14], grasp assessment [15] [16], and pose detection [17]. Beside the visual and tactile data fusion, the compound concerning visual and force/torque feedbacks is also under study.…”

Section: A Multi-modal Sensing In Manipulationmentioning

confidence: 99%

Multi-Modal Fusion in Contact-Rich Precise Tasks via Hierarchical Policy Learning

Jin¹,

Lin²,

Tan³

et al. 2022

Preprint

View full text Add to dashboard Cite

Combined visual and force feedback play an essential role in contact-rich robotic manipulation tasks. Current methods focus on developing the feedback control around a single modality while underrating the synergy of the sensors. Fusing different sensor modalities is necessary but remains challenging. A key challenge is to achieve an effective multimodal and generalized control scheme to novel objects with precision. This paper proposes a practical multi-modal sensor fusion mechanism using hierarchical policy learning. To begin with, we use a self-supervised encoder that extracts multi-view visual features and a hybrid motion/force controller that regulates force behaviors. Next, the multi-modality fusion is simplified by hierarchical integration of the vision, force, and proprioceptive data in the reinforcement learning (RL) algorithm. Moreover, with hierarchical policy learning, the control scheme can exploit the visual feedback limits and explore the contribution of individual modality in precise tasks. Experiments indicate that robots with the control scheme could assemble objects with 0.25mm clearance in simulation. The system could be generalized to widely varied initial configurations and new shapes. Experiments validate that the simulated system can be robustly transferred to reality without fine-tuning.

show abstract

“…The position of the collision is used to determine the pose using a particle filter. This approach has also been extended with a visual check [15]. However, this approach is not feasible for our system as the collisions with the table can bend the pins.…”

Section: A Pose Estimationmentioning

confidence: 99%

In-Hand Pose Estimation and Pin Inspection for Insertion of Through-Hole Components

Hagelskjær¹,

Kraft²

2022

Preprint

View full text Add to dashboard Cite

The insertion of through-hole components is a difficult task. As the tolerances of the holes are very small, minor errors in the insertion will result in failures. These failures can damage components and will require manual intervention for recovery. Errors can occur both from imprecise object grasps and bent pins. Therefore, it is important that a system can accurately determine the object's position and reject components with bent pins.By utilizing the constraints inherent in the object grasp a method using template matching is able to obtain very precise pose estimates. Methods for pin-checking are also implemented, compared, and a successful method is shown.The set-up is performed automatically, with two novel contributions. A deep learning segmentation of the pins is performed and the inspection pose is found by simulation. From the inspection pose and the segmented pins, the templates for pose estimation and pin check are then generated.To train the deep learning method a dataset of segmented through-hole components is created. The network shows a 97.3 % accuracy on the test set. The pin-segmentation network is also tested on the insertion CAD models and successfully segment the pins.The complete system is tested on three different objects, and experiments show that the system is able to insert all objects successfully. Both by correcting in-hand grasp errors and rejecting objects with bent pins.

show abstract

Precise Multi-Modal In-Hand Pose Estimation using Low-Precision Sensors for Robotic Assembly

Cited by 28 publications

References 40 publications

Uncertainty-Aware Manipulation Planning Using Gravity and Environment Geometry

Uncertainty-Aware Manipulation Planning Using Gravity and Environment Geometry

Multi-Modal Fusion in Contact-Rich Precise Tasks via Hierarchical Policy Learning

In-Hand Pose Estimation and Pin Inspection for Insertion of Through-Hole Components

Contact Info

Product

Resources

About