Traffic Engineering and Characterization of High-Rate Large-Sized Flows

Abstract

High-rate large-sized (α) flows have adverse effects on delay-sensitive flows. Research-andeducation network providers are interested in identifying such flows within their networks, and directing these flows to virtual circuits. To achieve this goal, a design was proposed for a hybrid network traffic engineering system (HNTES) that would run on an external server, gather NetFlow records from routers, analyze these records to identify α-flow source/destination address prefixes, configure firewall filter rules at ingress routers to extract future α flows and redirect them to provisioned virtual circuits. This thesis presents an evaluation of this HNTES design using NetFlow records collected over a 7-month period from four ESnet routers. The results show that the HNTES effectiveness was above 90% for NetFlow records collected at edge routers, which corresponded to file downloads from Department of Energy (DOE) laboratories, while the effectiveness was lower for peering routers whose NetFlow records corresponded to file uploads. With further investigation, we found that uploads were less frequent and involved fewer source/destination pairs than downloads.The thesis also describes an algorithm for characterizing the size, duration, average rate, and frequency of α flows, from NetFlow records. The algorithm was validated using independently collected usage logs from application servers. This algorithm can be used in a network management system for providers interested in these types of flows, such as research-and-education network providers whose customers move large scientific datasets. We executed the algorithm on the same NetFlow records used in the HNTES evaluation. Flows moving datasets as large as 811 GB and at rates as high as 5.7 Gbps were observed. Some source-destination pairs were found to repeatedly create α flows. An analysis of the rates of the 1596 repeated α flows created by one pair ii iii showed considerable variance, with minimum rate of 100 Mbps, maximum rate of 536 Mbps, and a coefficient of variation of 30%.

Characterization of high-rate large-sized flows

“…Our validation showed high accuracy for both size and duration of high-rate, large-sized flows for selected thresholds [8]. Our algorithm for size/duration computation is similar to that used by Fioreze et al [7] with one difference.…”

“…Given our focus on designing network management systems and not new router hardware, our scheme relies on built-in NetFlow/IPFIX systems supported in most deployed provider routers. Our prior papers [6], [16] proposed a solution for traffic engineering α flows, but did not characterize the size, duration and rate of these flows, which is the contribution of this paper.…”

“…There is interest in characterizing α flows for various applications, such as traffic engineering α flows to optical circuits or MultiProtocol Label Switched (MPLS) paths to isolate them from general-purpose flows [5], [6], identifying BGP misconfigurations that could cause α flows to enter on less-desirable peering links, and identifying performance issues such as high throughput variance.…”

Machine learning in eigensubspace for network path identification and flow forecast

“…Recent developments in the analysis of network traffic data have mostly focused on classification and characterization of flows by using the flow level statistics [1–9]. Principal Component Analysis (PCA) was used to analyze the IP flow statistics to identify anomalous behaviour in the network [10].…”