Unbiased experiments in congested networks

Abstract

When developing a new networking algorithm, it is established practice to run a randomized experiment, or A/B test, to evaluate its performance. In an A/B test, traffic is randomly allocated between a treatment group, which uses the new algorithm, and a control group, which uses the existing algorithm. However, because networks are congested, both treatment and control traffic compete against each other for resources in a way that biases the outcome of these tests. This bias can have a surprisingly large effect; for example, in lab A/B tests with two widely used congestion control algorithms, the treatment appeared to deliver 150% higher throughput when used by a few flows, and 75% lower throughput when used by most flowsdespite the fact that the two algorithms have identical throughput when used by all traffic.Beyond the lab, we show that A/B tests can also be biased at scale. In an experiment run in cooperation with Netflix, estimates from A/B tests mistake the direction of change of some metrics, miss changes in other metrics, and overestimate the size of effects. We propose alternative experiment designs, previously used in online platforms, to more accurately evaluate new algorithms and allow experimenters to better understand the impact of congestion on their tests.

High-Speed Path Probing Method for Large-Scale Network

“…The Internet has a complex network topology. On the one hand, the topology obtained from a single probing source is not comprehensive [ 28 , 29 , 30 , 31 ]. On the other hand, reducing the probing redundancy often means reducing the discovery of additional routes because of the load balancing phenomenon [ 32 ].…”