Towards Autonomous Robotic Assembly: Using Combined Visual and Tactile Sensing for Adaptive Task Execution

Nottensteiner, Korbinian; Sachtler, Arne; Albu‐Schäffer, Alin

doi:10.1007/s10846-020-01303-z

Cited by 17 publications

(11 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…State estimation has been applied to assembly tasks in prior works primarily using force data [5][6][7]. [8] and [9] fuse visual observations with force data to track part poses, but both methods rely on manually defined image features such as line or blob detectors. All these methods adopt particle filtering to estimate the hole position.…”

Section: B State Estimation For Assembly Tasksmentioning

confidence: 99%

“…State estimation methods seek to solve this problem by predicting the ground truth state of the environment from noisy observations [1][2][3][4]. A typical state representation is object poses [5][6][7][8][9], which are continuously estimated and then used to guide the robot's motion, e.g., for aligning the end-effector with a door knob. However, less attention has been paid to estimating high-level symbolic states, such as whether the door knob is locked or the door is fully shut.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Symbolic State Estimation with Predicates for Contact-Rich Manipulation Tasks

Migimatsu¹,

Lian²,

Bohg³

et al. 2022

Preprint

View full text Add to dashboard Cite

Manipulation tasks often require a robot to adjust its sensorimotor skills based on the state it finds itself in. Taking peg-in-hole as an example: once the peg is aligned with the hole, the robot should push the peg downwards. While high level execution frameworks such as state machines and behavior trees are commonly used to formalize such decision-making problems, these frameworks require a mechanism to detect the high-level symbolic state. Handcrafting heuristics to identify symbolic states can be brittle, and using data-driven methods can produce noisy predictions, particularly when working with limited datasets, as is common in real-world robotic scenarios. This paper proposes a Bayesian state estimation method to predict symbolic states with predicate classifiers. This method requires little training data and allows fusing noisy observations from multiple sensor modalities. We evaluate our framework on a set of real-world peg-in-hole and connector-socket insertion tasks, demonstrating its ability to classify symbolic states and to generalize to unseen tasks, outperforming baseline methods.We also demonstrate the ability of our method to improve the robustness of manipulation policies on a real robot.

show abstract

Section: B State Estimation For Assembly Tasksmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Symbolic State Estimation with Predicates for Contact-Rich Manipulation Tasks

Migimatsu¹,

Lian²,

Bohg³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This research includes dimension inspection [ 30 ], object recognition [ 31 ], and localization [ 29 ]. In the context of robot assembly, visual and tactile sensing has been used to continuously track assembly parts using multimodal fusion based on particle filters [ 32 ] and Bayesian state estimation [ 13 ]. We share the principal idea of combining data from visual and force-based sensing.…”

Section: Related Workmentioning

confidence: 99%

“…The evaluation of the proposed approach comprised two scenarios. Peg-in-hole assembly is chosen as the first use case because it reflects the typical complexity of industrial assembly tasks [ 13 ]. We show that it is possible to apply the method to other tasks by performing an evaluation of a car starter assembly task.…”

Section: Introductionmentioning

confidence: 99%

Determining Exception Context in Assembly Operations from Multimodal Data

Simonič

Hrovat

Džeroski

et al. 2022

Sensors

View full text Add to dashboard Cite

Robot assembly tasks can fail due to unpredictable errors and can only continue with the manual intervention of a human operator. Recently, we proposed an exception strategy learning framework based on statistical learning and context determination, which can successfully resolve such situations. This paper deals with context determination from multimodal data, which is the key component of our framework. We propose a novel approach to generate unified low-dimensional context descriptions based on image and force-torque data. For this purpose, we combine a state-of-the-art neural network model for image segmentation and contact point estimation using force-torque measurements. An ensemble of decision trees is used to combine features from the two modalities. To validate the proposed approach, we have collected datasets of deliberately induced insertion failures both for the classic peg-in-hole insertion task and for an industrially relevant task of car starter assembly. We demonstrate that the proposed approach generates reliable low-dimensional descriptors, suitable as queries necessary in statistical learning.

show abstract

“…However, a pre-constructed object model is required, which is difficult for the previously mentioned operating object. In the field of robotic assembly, Korbinian et al [ 22 ] presented a framework for tracking visual and tactile information assembly to perform assembly operations for multiple hole types. Besides, some researchers have fused haptic and visual information using a Bayesian framework to achieve the estimation of target positions of assembled objects in industrial assembly [ 23 , 24 ].…”

Section: Introductionmentioning

confidence: 99%

Multi-Sensor Perception Strategy to Enhance Autonomy of Robotic Operation for Uncertain Peg-in-Hole Task

Qin

Wang

Yuan

et al. 2021

Sensors

View full text Add to dashboard Cite

The peg-in-hole task with object feature uncertain is a typical case of robotic operation in the real-world unstructured environment. It is nontrivial to realize object perception and operational decisions autonomously, under the usual visual occlusion and real-time constraints of such tasks. In this paper, a Bayesian networks-based strategy is presented in order to seamlessly combine multiple heterogeneous senses data like humans. In the proposed strategy, an interactive exploration method implemented by hybrid Monte Carlo sampling algorithms and particle filtering is designed to identify the features' estimated starting value, and the memory adjustment method and the inertial thinking method are introduced to correct the target position and shape features of the object respectively. Based on the Dempster–Shafer evidence theory (D-S theory), a fusion decision strategy is designed using probabilistic models of forces and positions, which guided the robot motion after each acquisition of the estimated features of the object. It also enables the robot to judge whether the desired operation target is achieved or the feature estimate needs to be updated. Meanwhile, the pliability model is introduced into repeatedly perform exploration, planning and execution steps to reduce interaction forces, the number of exploration. The effectiveness of the strategy is validated in simulations and in a physical robot task.

show abstract

Towards Autonomous Robotic Assembly: Using Combined Visual and Tactile Sensing for Adaptive Task Execution

Cited by 17 publications

References 64 publications

Symbolic State Estimation with Predicates for Contact-Rich Manipulation Tasks

Symbolic State Estimation with Predicates for Contact-Rich Manipulation Tasks

Determining Exception Context in Assembly Operations from Multimodal Data

Multi-Sensor Perception Strategy to Enhance Autonomy of Robotic Operation for Uncertain Peg-in-Hole Task

Contact Info

Product

Resources

About