With the wide deployment of flow monitoring in IP networks, the analysis of the exported flow data has become an important research area. It has been shown that flow data can be used to detect traffic anomalies, DoS attacks, and the propagation of worms. In practice, anomalies and attacks should be detected as fast as possible in order to allow taking appropriate countermeasures. We describe the necessary steps from the raw flow data to the detection result in a systematic way. Furthermore, we present TOPAS, a system and framework for real-time analysis of flow data, that has been developed in order to meet these requirements. Performance measurements and various application examples point out the capabilities and benefits of our approach.

Abstract. This paper presents a novel traffic classification approach which classifies TCP connections with help of observable Markov models. As traffic properties, payload length, direction, and position of the first packets of a TCP connection are considered. We evaluate the accuracy of the classification approach with help of packet traces captured in a real network, achieving higher accuracies than the cluster-based classification approach of Bernaille [1]. As another advantage, the complexity of the proposed Markov classifier is low for both training and classification. Furthermore, the classification approach provides a certain level of robustness against changed usage of applications.

Malware and botnets pose a steady and growing threat to network security. Therefore, packet analysis systems examine network traffic to detect active botnets and spreading worms. However, with the advent of multi-gigabit link speeds, capturing and analysing header and payload of every packet requires enormous amounts of computational resources and is therefore not feasible in many situations. We address this problem by presenting an efficient packet sampling algorithm that picks a small number of packets from the beginning of every TCP connection. Bloom filters are used to store the required connection state information with constant amount of memory. Our analysis of worm and botnet traffic shows that the large majority of attack signatures is actually found in these packets. Thus, our sampling algorithm can be deployed in front of a detection system to reduce the amount of inspected packets without degrading the detection results significantly.

Network monitoring is a major building block for many domains in communication networks. Besides typical accounting mechanisms and the emerging area of charging in next generation networks, especially network security solutions rely on efficient and optimized monitoring. Network monitoring in high-speed networks is usually based on flow accounting and aggregation techniques represent a necessary enhancement in order to cope with increasing amounts of monitoring data that accrue with the ever-growing network capacities. In this paper, we propose a flexible flow aggregation mechanism that can be directly employed on a monitoring probe to reduce the memory and processing demands. Alternatively, it can work as a concentrator that collects flow data from multiple monitoring probes, combines and aggregates them and forwards the results to an analyzer. We verified and evaluated the aggregation mechanism by integrating it into our monitoring probe Vermont. Our approach opens new prospects for high-speed network monitoring and allows coping with special situations that cannot be treated satisfyingly by traditional flow accounting, such as distributed denial-of-service attacks causing very high numbers of flows. Aggregated flow data are an easy-to-handle form of packet information especially for anomaly detection and accounting issues.

No abstract