Self-management of hybrid networks: Can we trust netflow data?

Abstract

Network measurement provides vital information on the health of managed networks. The collection of network information can be used for several reasons (e.g., accounting or security) depending on the purpose the collected data will be used for. At the University of Twente (UT), an automatic decision process for hybrid networks that relies on collected network information has been investigated. This approach, called selfmanagement of hybrid networks requires information retrieved from measuring processes in order to automatically decide on establishing/releasing lambda-connections for IP flows that are long in duration and big in volume (known as elephant flows). Nonetheless, the employed measurement technique can break the self-management decisions if the reported information does not accurately describe the actual behavior and characteristics of the observed flows. Within this context, this paper presents an investigation on the trustfulness of measurements performed using the popular NetFlow monitoring solution when elephant flows are especially observed. We primarily focus on the use of NetFlow with sampling in order to collect network information and investigate how reliable such information is for the selfmanagement processes. This is important because the selfmanagement approach decides which flows should be offloaded to the optical level based on the current state of the network and its running flows. We observe three specific flow metrics: octets, packets, and flow duration. Our analysis shows that NetFlow provides reliable information regarding octets and packets. On the other hand, the flow duration reported when sampling is employed tends to be shorter than the actual duration.

Characterization of high-rate large-sized flows

“…Our validation showed high accuracy for both size and duration of high-rate, large-sized flows for selected thresholds [8]. Our algorithm for size/duration computation is similar to that used by Fioreze et al [7] with one difference. We aggregate records into flows for only those flows that had at least one NetFlow record in which a size threshold of 1 GB was exceeded (since size computation from sampled NetFlow records is not accurate for smaller-sized/lower-rate flows).…”

“…Solution approach: Prior work has shown that NetFlow data can be used to accurately determine the size of large-sized, high-duration flows even with 1-in-1000 packet sampling rates though duration accuracy was reported to be lower [7]. Our validation showed high accuracy for both size and duration of high-rate, large-sized flows for selected thresholds [8].…”

“…Related work: Flow characterization studies include early work by Lan and Heidemann [10], above-described paper by Fioreze et al [7], a three-dimensional characterization of P2P, gaming and social-networking flows [11], an analysis of elephant flows [12], and a characterization for anomaly detection [13]. Bayesian methods have been proposed for identifying high-rate and/or large-sized flows but not characterizing flow sizes and rates [14].…”

Self-management of hybrid optical and packet switching networks

“…As a result of sampling, the collected network traffic is an estimation rather than the real traffic. Our goal in this section is to observe whether the use of sampling on elephant flows may inaccurately report our metrics of interest, namely number of bytes, number of packets, and duration [54]. These metrics are needed to calculate Bps and Pps throughputs.…”

Traffic Engineering in Packet/Circuit Hybrid Networks

“…Fioreze et al [52] proposed redirecting elephant flows (which they defined as long in duration and big in volume) to optical circuits in hybrid IP/optical networks. The focus of this work was on evaluating the impact of sampling rates.…”