Affordance-Based Grasping and Manipulation in Real World Applications

Abstract

In real world applications, robotic solutions remain impractical due to the challenges that arise in unknown and unstructured environments. To perform complex manipulation tasks in complex and cluttered situations, robots need to be able to identify the interaction possibilities with the scene, i. e. the affordances of the objects encountered. In unstructured environments with noisy perception, insufficient scene understanding and limited prior knowledge, this is a challenging task. In this work, we present an approach for grasping unknown objects in cluttered scenes with a humanoid robot in the context of a nuclear decommissioning task. Our approach combines the convenience and reliability of autonomous robot control with the precision and adaptability of teleoperation in a semi-autonomous selection of grasp affordances. Additionally, this allows exploiting the expert knowledge of an experienced human worker. To evaluate our approach, we conducted 75 real world experiments with more than 660 grasp executions on the humanoid robot ARMAR-6. The results demonstrate that high-level decisions made by the human operator, supported by autonomous robot control, contribute significantly to successful task execution.

Virtual Reality via Object Pose Estimation and Active Learning: Realizing Telepresence Robots with Aerial Manipulation Capabilities

“…AeroVR (Yashin et al, 2019) ARMAR-6 (Pohl et al, 2020) ModelSegmentation (Kohn et al, 2018) AvatarDrone (Kim and Oh, 2021) PaintCopter (Vempati et al, 2019) AR (Liu and Shen, 2020) AR (Puljiz et al, 2020) GraspLook (Ponomareva et al, 2021) The proposed system tasks in industrial settings. By executing over 70 executions of the aforementioned tasks over days and nights, from spring to winter, and with different users and locations, the benefits of our VR based telepresence concept are illustrated for enhancing aerial manipulation capabilities in real world industrial applications.…”

“…As the main challenge of reconstruction based methods is the limited bandwidth in communication, Kohn et al (2018) proposes an object recognition pipeline, i.e., replace the detected object with sparse virtual meshes and discard the dense sensor data. Pohl et al (2020) uses RGB-D sensor to construct a VR for affordance based manipulation with a humanoid, while Liu and Shen (2020) and Puljiz et al (2020) create augmented reality for a drone and a manipulator, respectively. Pace et al (2021) conducts a user study and argues that the point clouds of RGB-D sensors are noisy and inaccurate (with artifacts), which motivates for point cloud preprocessing methods for telepresence applications (Pace et al, 2021).…”

Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations

“…VGN [4] predicts 6-DoF grasps in clutter with a one-stage pipeline from input depth images. There is also a line of works that estimate affordance of an object or a scene first and then detect grasps based on estimated affordance [42,24,54]. In most of the prior works, deep networks are trained end-to-end with only grasp supervision.…”

A Memory System of a Robot Cognitive Architecture and its Implementation in ArmarX

“…Grasping is one of the most important abilities of a robotic system. Here, we consider the example of an affordance-based grasping pipeline as in [50]. In such a pipeline, an affordance extraction component first visually detects grasping affordances of an object or parts of the scene.…”