Experiments as Code: A Concept for Reproducible, Auditable, Debuggable, Reusable, & Scalable Experiments

Abstract

A common concern in experimental research is the auditability and reproducibility of experiments. Experiments are usually designed, provisioned, managed, and analyzed by diverse teams of specialists (e.g., researchers, technicians and engineers) and may require many resources (e.g. cloud infrastructure, specialized equipment). Even though researchers strive to document experiments accurately, this process is often lacking, making it hard to reproduce them. Moreover, when it is necessary to create a similar experiment, very often we end up "reinventing the wheel" as it is easier to start from scratch than trying to reuse existing work, thus losing valuable embedded best practices and previous experiences. In behavioral studies this has contributed to the reproducibility crisis. To tackle this challenge, we propose the "Experiments as Code" paradigm, where the whole experiment is not only documented but additionally the automation code to provision, deploy, manage, and analyze it is provided. To this end we define the Experiments as Code concept, provide a taxonomy for the components of a practical implementation, and provide a proof of concept with a simple desktop VR experiment that showcases the benefits of its "as code" representation, i.e., reproducibility, auditability, debuggability, reusability, & scalability.

The design, experiment, analyse, and reproduce principle for experimentation in virtual reality

“…The multiplication of frameworks can be historically grouped into three generations (Aguilar et al, 2022). In the first generation in the early 2000s, the framework mostly took care of the hardware (Tramberend, 1999;Allen et al, 2000;Ayaz et al, 2008;Annett and Bischof, 2009;Mossel et al, 2012).…”

“…The third-generation of frameworks is only emerging now and is shifting the focus away from the design to reproduction (Aguilar et al, 2022;Colombo et al, 2022;Colombo et al, 2023) under the guidance of preproducibility (Stark, 2018). Reproducibility has become the Achilles' heel of modern science, with several experiments not being able to continuously produce the same results (Ioannidis et al, 2015;Camerer et al, 2018) and consequently putting the claims made based on the experiments performed on shaky grounds (Collaboration, 2015).…”

“…What the second generation was still missing is the ability to quickly reproduce original experiments and possibly modify them to gain new insights and helping with reproducibility. The third generation helps provision the hardware for an experiment and deploy all the software, including the required background software for data collection, the experiment application, the data storage, the data management, and the data analysis through the Experiments as Code (ExaC) paradigm (Aguilar et al, 2022). Protocols about the experimental process and other documentations are explicitly part of the framework.…”

“…Platforms have a more general goal than frameworks and usually provide generalised services beyond the scope of a single type of application. Modern VR frameworks usually rely on Unity or Unreal game platforms as a basis and are compiled within the respective platforms and not in a stand-alone way (Aguilar et al, 2022).…”

“…Today, the number of frameworks is booming (Aguilar et al, 2022), and it has become more difficult than at any point in time before to decide which framework to use. VR frameworks are currently cannibalising each other as they are only marginally different but still crucially different for the users across different disciplines and sub-disciplines.…”