Inferring Functional Connectivity From Time-Series of Events in Large Scale Network Deployments

Abstract

To respond rapidly and accurately to network and service outages, network operators must deal with a large number of events resulting from the interaction of various services operating on complex, heterogeneous and evolving networks. In this paper, we introduce the concept of functional connectivity as an alternative approach to monitoring those events. Commonly used in the study of brain dynamics, functional connectivity is defined in terms of the presence of statistical dependencies between nodes. Although a number of techniques exist to infer functional connectivity in brain networks, their straightforward application to commercial network deployments is severely challenged by: (a) non-stationarity of the functional connectivity, (b) sparsity of the time-series of events, and (c) absence of an explicit model describing how events propagate through the network or indeed whether they propagate. Thus, in this paper, we present a novel inference approach whereby two nodes are defined as forming a functional edge if they emit substantially more coincident or short-lagged events than would be expected if they were statistically independent. The output of the method is an undirected weighted graph, where the weight of an edge between two nodes denotes the strength of the statistical dependence between them. We develop a model of time-varying functional connectivity whose parameters are determined by maximising the model's predictive power from one time window to the next. We assess the accuracy, efficiency and scalability of our method on two real datasets of network events spanning multiple months and on synthetic data for which ground truth is available. We compare our method against both a general-purpose time-varying network inference method and network management specific causal inference technique and discuss its merits in terms of sensitivity, accuracy and, importantly, scalability.Index Terms-Network management, network events, functional connectivity inference.

Exploiting Functional Connectivity Inference for Efficient Root Cause Analysis

“…However, such prior knowledge can be difficult to obtain in complex systems, such as modern network infrastructure. Our method for inferring functional connectivity [11] provides a highly scalable method for generating knowledge of statistical dependencies. Therefore, we propose to use the knowledge gained from inferred FC as prior-knowledge input for causal inference.…”

“…The approach provides a powerful method for developing insights into relationships between neurological components without knowledge of intricate underlying neuronal connectivity. In [11] we introduce a method for inferring FC from sparse and unreliable system log messages (see Section 2.1 below for brief summary). Unlike causal inference methods, inferred relationships here are purely correlative in nature and do not provide any insight into causal interactions.…”

“…In our method [11] pair-wise statistical dependence of device log activity is estimated by counting the number of times log events occurred within a given time lag σ for each pair of devices X and Y producing log messages in Fig. 1: Events are binned, with a 1 indicating the presence of at least one event and a 0 indicating the absence of any events in a specific time bin.…”

“…It has been demonstrated that the PC algorithm can be employed to extract operationally exploitable causal relationships from system log data [5]. However, due to the pair-wise conditional independence comparisons made by the algorithm with all possible conditioning sets, the PC algorithm can incur significant processing times [11].…”

“…Functional connectivity is widely employed within Neuroscience to understand relationships between remote brain regions in the absence of absolute topological ground-truth, by capturing statistical dependencies in their activity. To identify functional connectivities we rely on our work introduced in [11] to infer statistical dependencies within network infrastructure from unreliably time-stamped system logs. Constructing a preliminary graph based on such dependencies allows us to restrict the initial conditions of causal inference algorithms, reducing the number of considerations that must be made.…”