Apples, Oranges, and Testbeds

Abstract

Abstract-Research into wireless sensor networks is rapidly moving from simulations to realistic testbeds. The widely varying characteristics (e.g., radio hardware, #nodes, topology) of various testbeds raises concerns about the validity of results across different testbeds. This paper presents empirical data of an experiment involving one application (Surge), two routing protocols (MultiHop and MintRoute), and two testbeds (MoteLab and MistLab). The outcome is somewhat mixed. When increasing the data rate, congestion causes goodput to fall off in a similar fashion on both testbeds, which is good, but only when ignoring MintRoute ill-behaving for low rates on MoteLab, which is bad. Accounting for differences in communication hardware is necessary, but even then results should be taken with a grain of salt. This certainly holds for TOSSIM simulation results, since we found that they generally do not match those of physical testbeds.

Characterizing topological bottlenecks for data delivery in CTP using simulation-based stress testing with natural selection

“…In general, failure in uncontrolled deployments is rarely visible in fine detail to protocol developers: "We frequently failed to understand [..] performance results and could not determine who was to blame (i.e., the testbed characteristics, or the routing layer?)" [11]. The performance of certain deployments of precursors of CTP has been found to be puzzling and counter to intuition, and different deployments were found not to paint a unified picture of the protocol performance: "comparing [evaluation] results from different testbeds is much like comparing apples and oranges" [11].…”

“…[11]. The performance of certain deployments of precursors of CTP has been found to be puzzling and counter to intuition, and different deployments were found not to paint a unified picture of the protocol performance: "comparing [evaluation] results from different testbeds is much like comparing apples and oranges" [11]. A problem was found in a deployment related to the management of the routing table in unexpectedly dense networks, similar to our topological problem T 1 : "On the field about 70 nodes form a single cell around the gateway, which forces MintRoute to make a selection since it has room for only 16 nodes in its neighbor list.…”

Exploiting the Burstiness of Intermediate-Quality Wireless Links

“…We expect our approach to be more beneficial if integrated with routing protocols supporting high traffic rates. Moreover, the room for throughput improvement in a bandwidth limited system, like a sensornets, is very limited: Langendoen [23] reports a maximum link throughput of 3 KB/s for CC2420 without routing in TinyOS. Therefore, in addition to our primary goal of reducing the number of transmission, the throughput increase revealed in Figure 12 is a welcome improvement in a multihop sensornet.…”

The tradeoffs between data delivery ratio and energy costs in wireless sensor networks

“…[17]. Some WSN protocols which performed well in controlled environments had as low as 2% data delivery in the field [9,26].…”

“…WSN testing is often done on indoor WSN testbeds [4,14], which form well-connected networks unlikely to reproduce the type of communication interruptions encountered later in environmental deployments. Furthermore, "experimental results obtained on a single testbed are very difficult to generalize" [17]. Since worst-case scenarios are statistically rare events in the state-space of the problem, non-exhaustive methods, such as testbed analysis [14,22] or random testing, are likely to miss them.…”