Virtually the Same: Comparing Physical and Virtual Testbeds

Abstract

Network designers, planners, and security professionals increasingly rely on large-scale testbeds based on virtualization to emulate networks and make decisions about real-world deployments. However, there has been limited research on how well these virtual testbeds match their physical counterparts. Specifically, does the virtualization that these testbeds depend on actually capture real-world behaviors sufficiently well to support decisions?As a first step, we perform simple experiments on both physical and virtual testbeds to begin to understand where and how the testbeds differ. We set up a web service on one host and run ApacheBench against this service from a different host, instrumenting each system during these tests.We define an initial repeatable methodology (algorithm) to quantitatively compare physical and virtual testbeds. Specifically we compare the testbeds at three levels of abstraction: application, operating system (OS) and network. For the application level, we use the ApacheBench results. For OS behavior, we compare patterns of system call orderings using Markov chains. This provides a unique visual representation of the workload and OS behavior in our testbeds. We also drill down into read-system-call behaviors and show how at one level both systems are deterministic and identical, but as we move up in abstractions that consistency declines. Finally, we use packet captures to compare network behaviors and performance. We reconstruct flows and compare per-flow and per-experiment statistics.From these comparisons, we find that the behavior of the workload in the testbeds is similar but that the underlying processes to support it do vary. The low-level network behavior can vary quite widely in packetization depending on the virtual network driver. While these differences can be important, and knowing about them will help experiment designers, the core application and OS behaviors still represent similar processes.

Exploration of Multifidelity Approaches for Uncertainty Quantification in Network Applications

“…Physical testbeds are costly to build and maintain and may not be suitable for all types of tests. Related work has compared network emulation to physical testbeds to discover where and how they differ [5]. In future work, we could expand upon our levels of multifidelity to include results from a physical testbed (or even an operational network) as the highest fidelity.…”

“…For the minimega emulation, we leveraged the models that we constructed during previous work [5]. Since this is an exploratory study, we do not attempt to vary design parameters such as the virtual network interface type as done in the previous work.…”

Traffic Generation using Containerization for Machine Learning

“…Background traffic is usually generated in real-time from scripts executed on the virtual machine, which constrains both the amount and heterogeneity of the data. Current experiments show that it is possible to use large network of virtual machines for traffic generation [7,8], however we did not find any publicly available dataset of that category.…”

A Comprehensive Survey on Cyber-Physical Smart Grid Testbed Architectures: Requirements and Challenges

“…The Sandia National Laboratories also developed a multi-functional testbed which has been used for vulnerability analysis [79], validation of new topologies, hardware, controls, communication, and security of microgrid [131,132]. In another research study, the testbed was used to compare the performance of a virtual testbed to an actual physical system [133]. The North China Electric Power University (NCEPU) designed a multi-functional CP-SG testbed for various research application that ranges from vulnerability analysis, cybersecurity, to integration of different renewable energy resources [74,134].…”