Network Traffic Analysis With Query Driven Visualization SC 2005 HPC Analytics Results

Abstract

Our analytics challenge is is to identify, characterize, and visualize anomalous subsets of large collections of network connection data. We use a combination of HPC resources, advanced algorithms, and visualization techniques. To effectively and efficiently identify the salient portions of the data, we rely on a multi-stage workflow that includes data acquisition, summarization (feature extraction), novelty detection, and classification. Once these subsets of interest have been identified and automatically characterized, we use a state-of-the-art highdimensional query system to extract data subsets for interactive visualization. Our approach is equally useful for other large-data analysis problems where it is more practical to identify interesting subsets of the data for visualization than to render all data elements. By reducing the size of the rendering workload, we enable highly interactive and useful visualizations. As a result of this work we were able to analyze six months worth of data interactively with response times two orders of magnitude shorter than with conventional methods.Motivating Example: Figure 1 shows an example of thousands of network connections (white lines) that one of our automatic clustering methods identifies as being correlated. Each of the shaded regions delimits a hyperbox for subsequent high dimensional queries. Since the entire dataset may represent billions of connections, it is critical to have good methods for defining and executing subset queries over multiple dimensions. In the past year, on the order of 500 Terabytes (TB) crossed the boundary between the Internet and unclassified networks at LANL, NERSC, and LBNL. Data collection tools at each of these boundaries collect summary information of approximately 10 billion distinct connections, or 1 TB, per year. In addition, routerbased information saved for every subnet internal to the LANL unclassified network totals 46 billion records and 2.5 TB per year, representing several Petabytes of network traffic. Post-processing of this data for analysis and indexing can increase the size several times. In the past, this data could only be analyzed as a whole by large batch processing jobs or in small segments, usually 6 to 24 hours at a time. The raw data consists of summary information on each session, and consists of start time, duration, protocol, source and destination byte counts, packet counts, IP addresses, port numbers, and flags describing completeness of the connection. Subsequently, some of these fields are further decomposed (octets of IP addresses, for example) and statistical properties derived on a per-host basis.

Enabling Real-Time Querying of Live and Historical Stream Data

“…There are a number of ways to parallelize TelegraphCQ to handle the stream volume [22], or alternatively the system can use Data Triage [19] to trade off result accuracy for speed. In terms of handling historical data, FastBit has been shown to parallelize well [25]. Overall, we anticipate the system we propose can be parallelized effectively on a modest cluster of computers.…”

“…In an number of performance measurements, WAH compressed indices were shown to significantly outperform other indices [27,28]. Re-cently, FastBit has also been used in analysis of network traffic data and shown to be able to handle massive data sets [25,2]. This makes it a convenient choice for our prototype.…”

“…TelegraphCQ [5] is a streaming query processor with a flexible declarative language. FastBit, a bitmap index based query engine for append-only data, has previously demonstrated its effectiveness at searching over large amounts of historical networking data [2,25].…”

Slope-Dependent Rendering of Parallel Coordinates to Reduce Density Distortion and Ghost Clusters

“…Parallel coordinates plots (PCPs) [11] are a well-researched visualization technique for multi-dimensional data. Studies have shown that they can be learned easily by non-visualization experts [15,16] and used in practice in various domains like finance [1], traffic safety [7], and network analysis [17]. Compared to related techniques such as scatter plot matrices and projections, PCPs have the advantage to identify, explore, and understand patterns across multiple dimensions.…”

Indexing million of packets per second using GPUs

“…Recent work has identified compressed bitmap indexes as a very effective indexing technology for network traffic data [5,12,13]. Bitmap indexes are more compact in size than competitive approaches, such as tree-based indexes, and provide significant speedup over complex multiattribute queries.…”