On Mimicking the Effects of the Reality Gap with Simulation-Only Experiments

Abstract

One issue in the automatic design of control software for robot swarms is the so-called reality gap-the difference between reality and the simulation models used in the automatic design process. It is commonly understood that the reality gap manifests itself as a drop in performance when control software developed in simulation is used to control physical robots. Yet, often disregarded is the relative nature of this performance drop: the reality gap does not affect equally all instances of control software. Indeed, one might observe a rank inversion: control software A might perform better than control software B in simulation, but perform worse on robots. The possibility of rank inversion undermines any performance comparison made in simulation. It would thus seem the only way to assess control software is in robot experiments, which are costly and time consuming. We argue it is unnecessary to assume reality is more complex than simulation models for the effects of the reality gap to occur. Indeed, we show that performance drop and rank inversion can occur if one automatically designs control software in simulation using a model and then assesses it in simulation on another model-what we call a pseudo-reality. Our results suggest that an appropriately conceived pseudo-reality could be used to test automatically-generated control software for performance drop and rank inversion, without performing robot experiments.

Automatic Design of Communication-Based Behaviors for Robot Swarms

“…EvoStick was formally defined in [18] to serve as a yardstick in the study of Vanilla, but had been previously analyzed in [19]. It was subsequently included in other empirical studies [17,16,5,33]. EvoCom targets the e-puck platform, as it is modeled by RM 2-see Table 1.…”

Complexity Measures: Open Questions and Novel Opportunities in the Automatic Design and Analysis of Robot Swarms

“…Models of a relatively simple structure are characterized by high bias and tend to under-fit data, whilst those of a relatively complex ones are affected by a large variance and tend to overfit (Geman et al, 1992 ). As it has been recently shown, performance drop and rank inversion may be observed also in a simulation-only experiments (Ligot and Birattari, 2018 , 2019 ): that is, when behaviors are designed using a simulation model and tested using another one. This indicates that performance drop and rank inversion should not be ascribed to some unique relationship between reality and the simulation environment used in the design process, but rather to an intrinsic characteristic of a design method—i.e., the complexity of the control software produced: high complexity exposes a method to the risk of overfitting the simulation environment used in the design process.…”

Peer Review #2 of "Iterative improvement in the automatic modular design of robot swarms (v0.1)"

“…As the design process is stochastic, we repeat it 10 times for every design method, leading to 10 instances of control software per design method. Each of these 10 instances of control software is then assessed 10 times in the design context and 10 times in a pseudo-reality context (Ligot and Birattari, 2018) to obtain the performance results. The pseudo-reality context is, like the design context, a realistic simulation of the control software.…”