Topological Data Analysis for Enhancing Embedded Analytics for Enterprise Cyber Log Analysis and Forensics

Abstract

Forensic analysis of logs is one responsibility of an enterprise cyber defense team; inherently, this is a big data task with thousands of events possibly logged in minutes of activity. Logged events range from authorized users typing incorrect passwords to malignant threats. Log analysis is necessary to understand current threats, be proactive against emerging threats, and develop new firewall rules. This paper describes embedded analytics for log analysis, which incorporates five mechanisms: numerical, similarity, graph-based, graphical analysis, and interactive feedback. Topological Data Analysis (TDA) is introduced for log analysis with TDA providing novel graph-based similarity understanding of threats which additionally enables a feedback mechanism to further analyze log files. Using real-world firewall log data from an enterprise-level organization, our end-to-end evaluation shows the effective detection and interpretation of log anomalies via the proposed process, many of which would have otherwise been missed by traditional means.

The Effectiveness of Fitness Wearable Technologies: Applying Big-Data Techniques to Analyze User Experience and Perception

“…The Mapper algorithm has three main steps: (1) the placement of input data in one or more bins, (2) inter-and intra-clustering within and between bins, and (3) finally, the creation of a new graphical network where clusters act as vertices and interactions between clusters act as edges (Bihl, et al, 2020). Mapper's most popular Python implement is KeplerMapper, which is a part of the larger suite of libraries, scikit-tda (see (Saul & Tralie, 2019)) (van Veen, et al, 2019).…”

Topological Data Analysis for Anomaly Detection in Host-Based Logs

“…We also considered the spectrum of the graph Laplacian and the hypergraph Laplacian as features. 2 By considering the 0 and 1-simplices of the final simplicial complex in the filtration we obtain a graph. We compute the spectrum of its Laplacian, which we use as a feature vector.…”

“…Finally, we implemented the count vectorisation strategy discussed in Section 2.2 as our baseline [14]. This vector is commonly used in the literature for automated anomaly detection in logs, so is a suitable comparison for our techniques [2,7]. We refer to this baseline as 'counts'.…”