Semantic Interpretation of Structured Log Files

Abstract

Data from computer log files record traces of events involving user activity, applications, system software and network traffic. Logs are usually intended for diagnostic and debugging purposes, but their data can be extremely useful in system audits and forensic investigations. Logs created by intrusion detection systems, web servers, anti-virus and anti-malware systems, firewalls and network devices have information that can reconstruct the activities of malware or a malicious agent, help plan for remediation and prevent attacks by revealing probes or intrusions before damage has been done. While existing tools like Splunk can help analyze logs with known schemas, understanding log whose format is unfamiliar or associated with new device or custom application can be challenging. We describe a framework for analyzing logs and automatically generating a semantic description of their schema and content in RDF. The framework begins by normalizing the log into columns and rows using regular expression-based and dictionary-based classifiers. Leveraging our existing work on inferring the semantics of tables, we associate semantic types with columns and, when possible, map them to concepts in general knowledge-bases (e.g. DBpedia) and domain specific ones (e.g., Unified Cybersecurity Ontology). We link cell values to known type instances (e.g., an IP address) and suggest relationships between columns. Converting large and verbose log files into such semantic representations reveals their meaning and supports search, integration and reasoning over the data.

The SLOGERT Framework for Automated Log Knowledge Graph Construction

“…In terms of semantic KGs, existing approaches have focused either on structured log data only [31], or on tasks such as entity [20] and relation [37] extraction in unstructured log data. Whereas some of the extraction methods introduced in this context are similar to our approach, their focus is less on log representation, but on cybersecurity information more general (e.g., textual descriptions of attacks).…”

From logs to Stories: Human-Centred Data Mining for Cyber Threat Intelligence

“…Numerous studies have attempted to change the log structure into a rich format to improve the understanding. Nimbalkar et al [19] translated log files and added semantics keywords. The results are demonstrated in the semantic RDF linked data, which is a machine interpretable representation.…”

Comparison and Model of Compression Techniques for Smart Cloud Log File Handling

“…They are typically semi-structured, combining regular entry types (dates, times, hosts) with irregular user-defined messages. For a primer on application log structures and their semantic interpretation, which is also exploited by more recent compression algorithms, the work by Nimbalkar et al [7] explains the problem domain and offers an RDF-based solution that links to domain vocabularies.…”