General Robot Dynamics Learning and Gen2Real

Abstract

Acquiring dynamics is an essential topic in robot learning, but up-to-date methods, such as dynamics randomization, need to restart to check nominal parameters, generate simulation data, and train networks whenever they face different robots. To improve it, we novelly investigate general robot dynamics, its inverse models, and Gen2Real, which means transferring to reality. Our motivations are to build a model that learns the intrinsic dynamics of various robots and lower the threshold of dynamics learning by enabling an amateur to obtain robot models without being trapped in details. This paper achieves the "generality" by randomizing dynamics parameters, topology configurations, and model dimensions, which in sequence cover the property, the connection, and the number of robot links. A structure modified from GPT is applied to access the pretraining model of general dynamics. We also study various inverse models of dynamics to facilitate different applications. We step further to investigate a new concept, "Gen2Real", to transfer simulated, general models to physical, specific robots. Simulation and experiment results demonstrate the validity of the proposed models and method. 1

Survey on Multimodal Transformers for Robots

“…In Ref. [100], the authors proposed a pre-training decoder model for forward and inverse models of robots. The structure of the robot has various factors such as the length and number of links and torque values.…”

“…Decision transformer [96] Trajectory transformer [99] Miscellaneous models w/ pre-train lamBERT [89] Deeper DRL [91] CoBERL [93] Gen2Real [100] Pre-trained VLN Fig. 6.…”

Simple and Effective Multimodal Learning Based on Pre-Trained Transformer Models

“…The proposed model fused multimodal information by simply concatenating the outputs from the pre-trained transformer models; therefore, the number of modalities it can handle depends on the number of existing pre-trained models. For example, pre-trained transformer models exist for human poses [39], biological signals [40], videos [21], [41], [42], and robot dynamics [43]. Consequently, on combining these pre-trained models and using the transformer layers on top, as in the proposed model, the model can be easily fine-tuned to a multimodal task.…”