Self-Supervised Learning of Visual Servoing for Low-Rigidity Robots Considering Temporal Body Changes

Abstract

Recognition of the current state is indispensable for the operation of a robot. There are various states to be recognized, such as whether an elevator door is open or closed, whether an object has been grasped correctly, and whether the TV is turned on or off. Until now, these states have been recognized by programmatically describing the state of a point cloud or raw image, by annotating and learning images, by using special sensors, etc. In contrast to these methods, we apply Visual Question Answering (VQA) from a Pre-Trained Vision-Language Model (PTVLM) trained on a large-scale dataset, to such binary state recognition. This idea allows us to intuitively describe state recognition in language without any re-training, thereby improving the recognition ability of robots in a simple and general way. We summarize various techniques in questioning methods and image processing, and clarify their properties through experiments.

Learning-Based Wiping Behavior of Low-Rigidity Robots Considering Various Surface Materials and Task Definitions

“…To solve (1), we use a learnable input variable, parametric bias [8], [9], which can extract multiple attractor dynamics from various data, to deal with different surface materials. Parametric bias has been mainly used for imitation learning, but we are now applying it to predictive model learning [10]- [12]. To solve (2), we use a contact sensor, uSkin [13], which contains 24 3-axis sensors in an area of 31 mm × 51.5 mm.…”