Estimating Latent Processes on a Network From Indirect Measurements

Abstract

In a communication network, point-to-point traffic volumes over time are critical for designing protocols that route information efficiently and for maintaining security, whether at the scale of an Internet service provider or within a corporation. While technically feasible, the direct measurement of point-to-point traffic imposes a heavy burden on network performance and is typically not implemented. Instead, indirect aggregate traffic volumes are routinely collected. We consider the problem of estimating point-to-point traffic volumes, x t , from aggregate traffic volumes, y t , given information about the network routing protocol encoded in a matrix A. This estimation task can be reformulated as finding the solutions to a sequence of ill-posed linear inverse problems, y t = A x t , since the number of origin-destination routes of interest is higher than the number of aggregate measurements available.Here, we introduce a novel multilevel state-space model (SSM) of aggregate traffic volumes with realistic features. We implement a naïve strategy for estimating unobserved point-to-point traffic volumes from indirect measurements of aggregate traffic, based on particle filtering. We then develop a more efficient two-stage inference strategy that relies on model-based regularization: a simple model is used to calibrate regularization parameters that lead to efficient/scalable inference in the multilevel SSM. We apply our methods to corporate and academic networks, where we show that the proposed inference strategy outperforms existing approaches and scales to larger networks. We also design a simulation study to explore the factors that influence the performance. Our results suggest that model-based regularization may be an efficient strategy for inference in other complex multilevel models. Supplementary materials for this article are available online.

On Delay Tomography: Fast Algorithms and Spatially Dependent Models

“…The data collected in this way bear the information about the network, from which network characteristics can be obtained indirectly via statistical inference. The characteristics that have been estimated in this manner include delay distributions [2]- [10], origin-destination traffics [1], [4], [11]- [21], link-level loss rates [22]- [33], loss patterns [34] and network topology [35]. In this paper, we focus on delay tomography, which aims to estimate link-level delay distributions from end-to-end measurements.…”

Polytope samplers for network tomography

“…Polytope sampling has been studied for both continuous and integer‐valued traffic models. Airoldi & Blocker () made a major contribution in the former setting with their random direction algorithm. In essence this works because any two points in a convex set are connected by a line segment lying entirely within the set, and hence it is possible to explore by generating candidate values sampled along suitably generated line segments from the current flow pattern.…”

Fast parameter estimation in loss tomography for networks of general topology

“…Arrival orders and arrival times of the probes carry the information of the network, from which many network characteristics can be inferred statistically. Characteristics that have been estimated in this manner include link-level loss rates [2,3,4,5,6,7,8,9,10,11,12,13], delay distributions [14,15,16,17,18,19,20,21,22,23], origin-destination traffic [1,16,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38], loss patterns [39], and the network topology [40]. In this paper, we focus on the problem of estimating loss rates.…”