Packet-Level Telemetry in Large Datacenter Networks

Abstract

Debugging faults in complex networks often requires capturing and analyzing traffic at the packet level. In this task, datacenter networks (DCNs) present unique challenges with their scale, traffic volume, and diversity of faults. To troubleshoot faults in a timely manner, DCN administrators must a) identify affected packets inside large volume of traffic; b) track them across multiple network components; c) analyze traffic traces for fault patterns; and d) test or confirm potential causes. To our knowledge, no tool today can achieve both the specificity and scale required for this task.We present Everflow, a packet-level network telemetry system for large DCNs. Everflow traces specific packets by implementing a powerful packet filter on top of "match and mirror" functionality of commodity switches. It shuffles captured packets to multiple analysis servers using load balancers built on switch ASICs, and it sends "guided probes" to test or confirm potential faults. We present experiments that demonstrate Everflow's scalability, and share experiences of troubleshooting network faults gathered from running it for over 6 months in Microsoft's DCNs.

Hardware-Software Co-Design for Network Performance Measurement

“…Prior switch-based network measurement systems [18,19,32,23,40,22] are constrained by their raw data, sourced from limited measurement support in existing switches, e.g., NetFlow, match-action rules, and packet mirroring. More recently, Gupta et al [22] propose to partition monitoring queries between switches and a stream processor (e.g., Spark streaming [12]), iteratively refining the set of packets captured through match-action rules in the switch.…”

“…Delays in identifying and diagnosing network performance problems can severely affect service availability. As a result, both industry and academia have expended considerable effort in network measurement [3,6,11,18,26,32,40].…”

“…Today's line-rate switches (switches with 10-100 ports running at 10-100 Gbit/s) support very limited measurement in the form of sampling [3,11], counting [28,29,30,39], and packet capture [23,40]. These statistics are insufficient to compute performance metrics like latency or loss.…”

Every Timestamp Counts: Accurate Tracking of Network Latencies Using Reconcilable Difference Aggregator

“…Consequently, the sampled end to end latencies provide coarse-grained metrics, but have low fidelity unless the sampling rates are very high. For example, software bugs or faulty interfaces in switches may randomly produce failures on some packets according to the route in the network or packet header information [31]. Unfortunately, high sampling frequencies have high bandwidth cost, computing cost, and storage overhead.…”

Fault Localization in Large-Scale Network Policy Deployment

“…In the previous example, if 95 TCAM rules are impacted, the hit ratio is 0.95. The wide variation of hit ratio values can occur due to (1) switch TCAM overflow; (2) TCAM corruption [11] that causes bit errors on a specific field in a TCAM rule or across TCAM rules; and (3) software bugs [12] that modify object's value wrong at controller or switch agent. While the SCORE algorithm allows change of a threshold value to handle noisy input data, such a static mechanism helps little in solving the problem at hand, confirmed by our evaluation results in §VI.…”